diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index d9c21f2ae21..3b2a3e70a43 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -567,7 +567,8 @@ pep-0679.rst @pablogsal pep-0680.rst @encukou pep-0681.rst @jellezijlstra pep-0682.rst @mdickinson -pep-0683.rst @ericsnowcurrently +pep-0683.rst @ericsnowcurrently @eduardo-elizondo +pep-0684.rst @ericsnowcurrently # pep-0684.rst pep-0685.rst @brettcannon # ... diff --git a/pep-0684.rst b/pep-0684.rst new file mode 100644 index 00000000000..c7b1268cd1b --- /dev/null +++ b/pep-0684.rst @@ -0,0 +1,634 @@ +PEP: 684 +Title: A Per-Interpreter GIL +Author: Eric Snow +Discussions-To: python-dev@python.org +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 08-Mar-2022 +Python-Version: 3.11 +Post-History: 08-Mar-2022 +Resolution: + +.. XXX Split out an informational PEP with all the relevant info, + based on the "Consolidating Runtime Global State" section? + +Abstract +======== + +Since Python 1.5 (1997), CPython users can run multiple interpreters +in the same process. However, interpreters in the same process +have always shared a significant +amount of global state. This is a source of bugs, with a growing +impact as more and more people use the feature. Furthermore, +sufficient isolation would facilitate true multi-core parallelism, +where interpreters no longer share the GIL. The changes outlined in +this proposal will result in that level of interpreter isolation. + + +High-Level Summary +================== + +At a high level, this proposal changes CPython in the following ways: + +* stops sharing the GIL between interpreters, given sufficient isolation +* adds several new interpreter config options for isolation settings +* adds some public C-API for fine-grained control when creating interpreters +* keeps incompatible extensions from causing problems + +The GIL +------- + +The GIL protects concurrent access to most of CPython's runtime state. +So all that GIL-protected global state must move to each interpreter +before the GIL can. + +(In a handful of cases, other mechanisms can be used to ensure +thread-safe sharing instead, such as locks or "immortal" objects.) + +CPython Runtime State +--------------------- + +Properly isolating interpreters requires that most of CPython's +runtime state be stored in the ``PyInterpreterState`` struct. Currently, +only a portion of it is; the rest is found either in global variables +or in ``_PyRuntimeState``. Most of that will have to be moved. + +This directly coincides with an ongoing effort (of many years) to greatly +reduce internal use of C global variables and consolidate the runtime +state into ``_PyRuntimeState`` and ``PyInterpreterState``. +(See `Consolidating Runtime Global State`_ below.) That project has +`significant merit on its own `_ +and has faced little controversy. So, while a per-interpreter GIL +relies on the completion of that effort, that project should not be +considered a part of this proposal--only a dependency. + +Other Isolation Considerations +------------------------------ + +CPython's interpreters must be strictly isolated from each other, with +few exceptions. To a large extent they already are. Each interpreter +has its own copy of all modules, classes, functions, and variables. +The CPython C-API docs `explain further `_. + +.. _caveats: https://docs.python.org/3/c-api/init.html#bugs-and-caveats + +However, aside from what has already been mentioned (e.g. the GIL), +there are a couple of ways in which interpreters still share some state. + +First of all, some process-global resources (e.g. memory, +file descriptors, environment variables) are shared. There are no +plans to change this. + +Second, some isolation is faulty due to bugs or implementations that +did not take multiple interpreters into account. This includes +CPython's runtime and the stdlib, as well as extension modules that +rely on global variables. Bugs should be opened in these cases, +as some already have been. + +Depending on Immortal Objects +----------------------------- + +:pep:`683` introduces immortal objects as a CPython-internal feature. +With immortal objects, we can share any otherwise immutable global +objects between all interpreters. Consequently, this PEP does not +need to address how to deal with the various objects +`exposed in the public C-API `_. +It also simplifies the question of what to do about the builtin +static types. (See `Global Objects`_ below.) + +Both issues have alternate solutions, but everything is simpler with +immortal objects. If PEP 683 is not accepted then this one will be +updated with the alternatives. This lets us reduce noise in this +proposal. + + +Motivation +========== + +The fundamental problem we're solving here is a lack of true multi-core +parallelism (for Python code) in the CPython runtime. The GIL is the +cause. While it usually isn't a problem in practice, at the very least +it makes Python's multi-core story murky, which makes the GIL +a consistent distraction. + +Isolated interpreters are also an effective mechanism to support +certain concurrency models. :pep:`554` discusses this in more detail. + +Indirect Benefits +----------------- + +Most of the effort needed for a per-interpreter GIL has benefits that +make those tasks worth doing anyway: + +* makes multiple-interpreter behavior more reliable +* has led to fixes for long-standing runtime bugs that otherwise + hadn't been prioritized +* has been exposing (and inspiring fixes for) previously unknown runtime bugs +* has driven cleaner runtime initialization (:pep:`432`, :pep:`587`) +* has driven cleaner and more complete runtime finalization +* led to structural layering of the C-API (e.g. ``Include/internal``) +* also see `Benefits to Consolidation`_ below + +Furthermore, much of that work benefits other CPython-related projects: + +* performance improvements ("faster-cpython") +* pre-fork application deployment (e.g. Instagram) +* extension module isolation (see :pep:`630`, etc.) +* embedding CPython + +Existing Use of Multiple Interpreters +------------------------------------- + +The C-API for multiple interpreters has been used for many years. +However, until relatively recently the feature wasn't widely known, +nor extensively used (with the exception of mod_wsgi). + +In the last few years use of multiple interpreters has been increasing. +Here are some of the public projects using the feature currently: + +* `mod_wsgi `_ +* `OpenStack Ceph `_ +* `JEP `_ +* `Kodi `_ + +Note that, with :pep:`554`, multiple interpreter usage would likely +grow significantly (via Python code rather than the C-API). + +PEP 554 +------- + +:pep:`554` is strictly about providing a minimal stdlib module +to give users access to multiple interpreters from Python code. +In fact, it specifically avoids proposing any changes related to +the GIL. Consider, however, that users of that module would benefit +from a per-interpreter GIL, which makes PEP 554 more appealing. + + +Rationale +========= + +During initial investigations in 2014, a variety of possible solutions +for multi-core Python were explored, but each had its drawbacks +without simple solutions: + +* the existing practice of releasing the GIL in extension modules + * doesn't help with Python code +* other Python implementations (e.g. Jython, IronPython) + * CPython dominates the community +* remove the GIL (e.g. gilectomy, "no-gil") + * too much technical risk (at the time) +* Trent Nelson's "PyParallel" project + * incomplete; Windows-only at the time +* ``multiprocessing`` + + * too much work to make it effective enough; + high penalties in some situations (at large scale, Windows) + +* other parallelism tools (e.g. dask, ray, MPI) + * not a fit for the stdlib +* give up on multi-core (e.g. async, do nothing) + * this can only end in tears + +Even in 2014, it was fairly clear that a solution using isolated +interpreters did not have a high level of technical risk and that +most of the work was worth doing anyway. +(The downside was the volume of work to be done.) + + +Specification +============= + +As `summarized above `__, this proposal involves the +following changes, in the order they must happen: + +1. `consolidate global runtime state `_ + (including objects) into ``_PyRuntimeState`` +2. move nearly all of the state down into ``PyInterpreterState`` +3. finally, move the GIL down into ``PyInterpreterState`` +4. everything else + * add to the public C-API + * implement restrictions in ``ExtensionFileLoader`` + + * work with popular extension maintainers to help + with multi-interpreter support + +Per-Interpreter State +--------------------- + +The following runtime state will be moved to ``PyInterpreterState``: + +* all global objects that are not safely shareable (fully immutable) +* the GIL +* mutable, currently protected by the GIL +* mutable, currently protected by some other per-interpreter lock +* mutable, may be used independently in different interpreters +* all other mutable (or effectively mutable) state + not otherwise excluded below + +Furthermore, a number of parts of the global state have already been +moved to the interpreter, such as GC, warnings, and atexit hooks. + +The following state will not be moved: + +* global objects that are safely shareable, if any +* immutable, often ``const`` +* treated as immutable +* related to CPython's ``main()`` execution +* related to the REPL +* set during runtime init, then treated as immutable +* mutable, protected by some global lock +* mutable, atomic + +Note that currently the allocators (see ``Objects/obmalloc.c``) are shared +between all interpreters, protected by the GIL. They will need to move +to each interpreter (or a global lock will be needed). This is the +highest risk part of the work to isolate interpreters and may require +more than just moving fields down from ``_PyRuntimeState``. Some of +the complexity is reduced if CPython switches to a thread-safe +allocator like mimalloc. + +.. _proposed capi: + +C-API +----- + +The following private API will be made public: + +* ``_PyInterpreterConfig`` +* ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``) + +The following fields will be added to ``PyInterpreterConfig``: + +* ``own_gil`` - (bool) create a new interpreter lock + (instead of sharing with the main interpreter) +* ``strict_extensions`` - fail import in this interpreter for + incompatible extensions (see `Restricting Extension Modules`_) + +Restricting Extension Modules +----------------------------- + +Extension modules have many of the same problems as the runtime when +state is stored in global variables. :pep:`630` covers all the details +of what extensions must do to support isolation, and thus safely run in +multiple interpreters at once. This includes dealing with their globals. + +Extension modules that do not implement isolation will only run in +the main interpreter. In all other interpreters, the import will +raise ``ImportError``. This will be done through +``importlib._bootstrap_external.ExtensionFileLoader``. + +We will work with popular extensions to help them support use in +multiple interpreters. This may involve adding to CPython's public C-API, +which we will address on a case-by-case basis. + +Extension Module Compatibility +'''''''''''''''''''''''''''''' + +As noted in `Extension Modules`_, many extensions work fine in multiple +interpreters without needing any changes. The import system will still +fail if such a module doesn't explicitly indicate support. At first, +not many extension modules will, so this is a potential source +of frustration. + +We will address this by adding a context manager to temporarily disable +the check on multiple interpreter support: +``importlib.util.allow_all_extensions()``. + +Documentation +------------- + +The "Sub-interpreter support" section of ``Doc/c-api/init.rst`` will be +updated with the added API. + + +Impact +====== + +Backwards Compatibility +----------------------- + +No behavior or APIs are intended to change due to this proposal, +with one exception noted in `the next section `_. +The existing C-API for managing interpreters will preserve its current +behavior, with new behavior exposed through new API. No other API +or runtime behavior is meant to change, including compatibility with +the stable ABI. + +See `Objects Exposed in the C-API`_ below for related discussion. + +Extension Modules +''''''''''''''''' + +Currently the most common usage of Python, by far, is with the main +interpreter running by itself. This proposal has zero impact on +extension modules in that scenario. Likewise, for better or worse, +there is no change in behavior under multiple interpreters created +using the existing ``Py_NewInterpreter()``. + +Keep in mind that some extensions already break when used in multiple +interpreters, due to keeping module state in global variables. They +may crash or, worse, experience inconsistent behavior. That was part +of the motivation for :pep:`630` and friends, so this is not a new +situation nor a consequence of this proposal. + +In contrast, when the `proposed API `_ is used to +create multiple interpreters, the default behavior will change for +some extensions. In that case, importing an extension will fail +(outside the main interpreter) if it doesn't indicate support for +multiple interpreters. For extensions that already break in +multiple interpreters, this will be an improvement. + +Now we get to the break in compatibility mentioned above. Some +extensions are safe under multiple interpreters, even though they +haven't indicated that. Unfortunately, there is no reliable way for +the import system to infer that such an extension is safe, so +importing them will still fail. This case is addressed in +`Extension Module Compatibility`_ below. + +Extension Module Maintainers +---------------------------- + +One related consideration is that a per-interpreter GIL will likely +drive increased use of multiple interpreters, particularly if :pep:`554` +is accepted. Some maintainers of large extension modules have expressed +concern about the increased burden they anticipate due to increased +use of multiple interpreters. + +Specifically, enabling support for multiple interpreters will require +substantial work for some extension modules. To add that support, +the maintainer(s) of such a module (often volunteers) would have to +set aside their normal priorities and interests to focus on +compatibility (see :pep:`630`). + +Of course, extension maintainers are free to not add support for use +in multiple interpreters. However, users will increasingly demand +such support, especially if the feature grows +in popularity. + +Either way, the situation can be stressful for maintainers of such +extensions, particularly when they are doing the work in their spare +time. The concerns they have expressed are understandable, and we address +the partial solution in `Restricting Extension Modules`_ below. + +Alternate Python Implementations +-------------------------------- + +Other Python implementation are not required to provide support for +multiple interpreters in the same process (though some do already). + +Security Implications +--------------------- + +There is no known impact to security with this proposal. + +Maintainability +--------------- + +On the one hand, this proposal has already motivated a number of +improvements that make CPython *more* maintainable. That is expected +to continue. On the other hand, the underlying work has already +exposed various pre-existing defects in the runtime that have had +to be fixed. That is also expected to continue as multiple interpreters +receive more use. Otherwise, there shouldn't be a significant impact +on maintainability, so the net effect should be positive. + +Performance +----------- + +The work to consolidate globals has already provided a number of +improvements to CPython's performance, both speeding it up and using +less memory, and this should continue. Performance benefits to a +per-interpreter GIL have not been explored. At the very least, it is +not expected to make CPython slower (as long as interpreters are +sufficiently isolated). + + +How to Teach This +================= + +This is an advanced feature for users of the C-API. There is no +expectation that this will be taught. + +That said, if it were taught then it would boil down to the following: + + In addition to Py_NewInterpreter(), you can use Py_NewInterpreterEx() + to create an interpreter. The config you pass it indicates how you + want that interpreter to behave. + + +Reference Implementation +======================== + + + + +Open Issues +=========== + +* What are the risks/hurdles involved with moving the allocators? +* Is ``allow_all_extensions`` the best name for the context manager? + + +Deferred Functionality +====================== + +* ``PyInterpreterConfig`` option to always run the interpreter in a new thread +* ``PyInterpreterConfig`` option to assign a "main" thread to the interpreter + and only run in that thread + + +Rejected Ideas +============== + + + + +Extra Context +============= + +Sharing Global Objects +---------------------- + +We are sharing some global objects between interpreters. +This is an implementation detail and relates more to +`globals consolidation `_ +than to this proposal, but it is a significant enough detail +to explain here. + +The alternative is to share no objects between interpreters, ever. +To accomplish that, we'd have to sort out the fate of all our static +types, as well as deal with compatibility issues for the many objects +`exposed in the public C-API `_. + +That approach introduces a meaningful amount of extra complexity +and higher risk, though prototyping has demonstrated valid solutions. +Also, it would likely result in a performance penalty. + +`Immortal objects `_ allow us to +share the otherwise immutable global objects. That way we avoid +the extra costs. + +.. _capi objects: + +Objects Exposed in the C-API +'''''''''''''''''''''''''''' + +The C-API (including the limited API) exposes all the builtin types, +including the builtin exceptions, as well as the builtin singletons. +The exceptions are exposed as ``PyObject *`` but the rest are exposed +as the static values rather than pointers. This was one of the few +non-trivial problems we had to solve for per-interpreter GIL. + +With immortal objects this is a non-issue. + + +Consolidating Runtime Global State +---------------------------------- + +As noted in `CPython Runtime State`_ above, there is an active effort +(separate from this PEP) to consolidate CPython's global state into the +``_PyRuntimeState`` struct. Nearly all the work involves moving that +state from global variables. The project is particularly relevant to +this proposal, so below is some extra detail. + +Benefits to Consolidation +''''''''''''''''''''''''' + +Consolidating the globals has a variety of benefits: + +* greatly reduces the number of C globals (best practice for C code) +* the move draws attention to runtime state that is unstable or broken +* encourages more consistency in how runtime state is used +* makes multiple-interpreter behavior more reliable +* leads to fixes for long-standing runtime bugs that otherwise + haven't been prioritized +* exposes (and inspires fixes for) previously unknown runtime bugs +* facilitates cleaner runtime initialization and finalization +* makes it easier to discover/identify CPython's runtime state +* makes it easier to statically allocate runtime state in a consistent way +* better memory locality for runtime state +* structural layering of the C-API (e.g. ``Include/internal``) + +Furthermore, much of that work benefits other CPython-related projects: + +* performance improvements ("faster-cpython") +* pre-fork application deployment (e.g. Instagram) +* extension module isolation (see :pep:`630`, etc.) +* embedding CPython + +Scale of Work +''''''''''''' + +The number of global variables to be moved is large enough to matter, +but most are Python objects that can be dealt with in large groups +(like ``Py_IDENTIFIER``). In nearly all cases, moving these globals +to the interpreter is highly mechanical. That doesn't require +cleverness but instead requires someone to put in the time. + +State To Be Moved +''''''''''''''''' + +The remaining global variables can be categorized as follows: + +* global objects + * static types (incl. exception types) + * non-static types (incl. heap types, structseq types) + * singletons (static) + * singletons (initialized once) + * cached objects +* non-objects + * will not (or unlikely to) change after init + * only used in the main thread + * initialized lazily + * pre-allocated buffers + * state + +Those globals are spread between the core runtime, the builtin modules, +and the stdlib extension modules. + +For a breakdown of the remaining globals, run: + +.. code-block:: bash + + ./python Tools/c-analyzer/table-file.py Tools/c-analyzer/cpython/globals-to-fix.tsv + +Already Completed Work +'''''''''''''''''''''' + +As mentioned, this work has been going on for many years. Here are some +of the things that have already been done: + +* cleanup of runtime initialization (see :pep:`432` / :pep:`587`) +* extension module isolation machinery (see :pep:`384` / :pep:`3121` / :pep:`489`) +* isolation for many builtin modules +* isolation for many stdlib extension modules +* addition of ``_PyRuntimeState`` +* no more ``_Py_IDENTIFIER()`` +* statically allocated: + + * empty string + * string literals + * identifiers + * latin-1 strings + * length-1 bytes + * empty tuple + +Tooling +''''''' + +As already indicated, there are several tools to help identify the +globals and reason about them. + +* ``Tools/c-analyzer/cpython/globals-to-fix.tsv`` - the list of remaining globals +* ``Tools/c-analyzer/c-analyzer.py`` + * ``analyze`` - identify all the globals + * ``check`` - fail if there are any unsupported globals that aren't ignored +* ``Tools/c-analyzer/table-file.py`` - summarize the known globals + +Also, the check for unsupported globals is incorporated into CI so that +no new globals are accidentally added. + +Global Objects +'''''''''''''' + +Global objects that are safe to be shared (without a GIL) between +interpreters can stay on ``_PyRuntimeState``. Not only must the object +be effectively immutable (e.g. singletons, strings), but not even the +refcount can change for it to be safe. Immortality (:pep:`683`) +provides that. (The alternative is that no objects are shared, which +adds significant complexity to the solution, particularly for the +objects `exposed in the public C-API `_.) + +Builtin static types are a special case of global objects that will be +shared. They are effectively immutable except for one part: +``__subclasses__`` (AKA ``tp_subclasses``). We expect that nothing +else on a builtin type will change, even the content +of ``__dict__`` (AKA ``tp_dict``). + +``__subclasses__`` for the builtin types will be dealt with by making +it a getter that does a lookup on the current ``PyInterpreterState`` +for that type. + + +References +========== + +Related: + +* :pep:`384` +* :pep:`432` +* :pep:`489` +* :pep:`554` +* :pep:`573` +* :pep:`587` +* :pep:`630` +* :pep:`683` +* :pep:`3121` + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.