Python 3.12 - ValueError: generator already executing

[sivel]

Bug description

It seems that with a combination of a commit to astroid, and a change in Python 3.12, I am getting a ValueError: generator already executing.

• pylint-dev/astroid@c5352d5
• python/cpython@6968f9e

Unfortunately, despite about 12 hours of trying, I don't have a simple reproducer, and this will involve running pylint against https://github.com/ansible/ansible

I've logged here instead of directly against astroid, as I was not able to find an easy way to cause the error with astroid alone.

Configuration

N/A

Command used

git clone git@github.com:ansible/ansible.git
cd ansible
python3.12 -m pylint lib/ansible/playbook/base.py > /dev/null

Pylint output

Traceback (most recent call last):
  File ".../astroid/astroid/decorators.py", line 104, in inner
    yield from generator
ValueError: generator already executing

If I insert additional debugging into astroid, to use traceback.print_stack() when that ValueError happens, I see something like:

Traceback...

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File ".../lib/python3.12/site-packages/pylint/__main__.py", line 10, in <module>
    pylint.run_pylint()
  File ".../lib/python3.12/site-packages/pylint/__init__.py", line 34, in run_pylint
    PylintRun(argv or sys.argv[1:])
  File ".../lib/python3.12/site-packages/pylint/lint/run.py", line 211, in __init__
    linter.check(args)
  File ".../lib/python3.12/site-packages/pylint/lint/pylinter.py", line 704, in check
    self._lint_files(ast_per_fileitem, check_astroid_module)
  File ".../lib/python3.12/site-packages/pylint/lint/pylinter.py", line 752, in _lint_files
    self._lint_file(fileitem, module, check_astroid_module)
  File ".../lib/python3.12/site-packages/pylint/lint/pylinter.py", line 788, in _lint_file
    check_astroid_module(module)
  File ".../lib/python3.12/site-packages/pylint/lint/pylinter.py", line 1017, in check_astroid_module
    retval = self._check_astroid_module(
  File ".../lib/python3.12/site-packages/pylint/lint/pylinter.py", line 1069, in _check_astroid_module
    walker.walk(node)
  File ".../lib/python3.12/site-packages/pylint/utils/ast_walker.py", line 94, in walk
    self.walk(child)
  File ".../lib/python3.12/site-packages/pylint/utils/ast_walker.py", line 94, in walk
    self.walk(child)
  File ".../lib/python3.12/site-packages/pylint/utils/ast_walker.py", line 94, in walk
    self.walk(child)
  File ".../lib/python3.12/site-packages/pylint/utils/ast_walker.py", line 91, in walk
    callback(astroid)
  File ".../lib/python3.12/site-packages/pylint/checkers/typecheck.py", line 1440, in visit_call
    called, implicit_args, callable_name = _determine_callable(called)
  File ".../lib/python3.12/site-packages/pylint/checkers/typecheck.py", line 636, in _determine_callable
    new = callable_obj.local_attr("__new__")[-1]
  File ".../astroid/astroid/nodes/scoped_nodes/scoped_nodes.py", line 2327, in local_attr
    class_node = next(self.local_attr_ancestors(name, context), None)
  File ".../astroid/astroid/nodes/scoped_nodes/scoped_nodes.py", line 2279, in local_attr_ancestors
    ancestors: Iterable[ClassDef] = self.mro(context)[1:]
  File ".../astroid/astroid/nodes/scoped_nodes/scoped_nodes.py", line 2924, in mro
    return self._compute_mro(context=context)
  File ".../astroid/astroid/nodes/scoped_nodes/scoped_nodes.py", line 2893, in _compute_mro
    inferred_bases = list(self._inferred_bases(context=context))
  File ".../astroid/astroid/nodes/scoped_nodes/scoped_nodes.py", line 2873, in _inferred_bases
    baseobj = next(
  File ".../astroid/astroid/decorators.py", line 56, in wrapped
    yield res
  File ".../astroid/astroid/bases.py", line 181, in _infer_stmts
    yield inf
  File ".../astroid/astroid/nodes/node_ng.py", line 175, in infer
    yield result
  File ".../astroid/astroid/decorators.py", line 56, in wrapped
    yield res
  File ".../astroid/astroid/bases.py", line 181, in _infer_stmts
    yield inf
  File ".../astroid/astroid/nodes/node_ng.py", line 175, in infer
    yield result
  File ".../astroid/astroid/decorators.py", line 56, in wrapped
    yield res
  File ".../astroid/astroid/bases.py", line 181, in _infer_stmts
    yield inf
  File ".../astroid/astroid/nodes/node_ng.py", line 175, in infer
    yield result
  File ".../astroid/astroid/decorators.py", line 56, in wrapped
    yield res
  File ".../astroid/astroid/nodes/node_ng.py", line 175, in infer
    yield result
  File ".../astroid/astroid/decorators.py", line 107, in inner
    traceback.print_stack(file=sys.stderr)

Expected behavior

No traceback

Pylint version

astroid 3.0.0
pylint 3.0.1
Python 3.12.0

OS / Environment

N/A

Additional dependencies

N/A

[sivel]

Based on my very non-expert research, it seems that due to using cls.is_subclass_of('enum.Enum') has caused .ancestors() to be called and .infer(), and that is somehow interfering with the ability to compute the mro in _compute_mro which also calls .infer(). As to exactly how, I'm at a little bit of a loss.

As to how exactly the cpython commit that I bisected earlier plays into this, I have no clue.

[jacobtylerwalls]

Thanks for the report. I've seen this from time to time. Do you have any sense of the severity of this issue?

[sivel]

This is preventing us from upgrading to pylint 3, as well as switching our default python version to py3.12. This not only affects the CI for ansible/ansible, but content authors using our ansible-test tooling.

It will also cause issues with the ability for Fedora (or other distros switching to python3.12 as their default) to package ansible, due to Fedora 39 needing to include pylint 3 as a result of their inclusion of Python 3.12. As such, ansible-test installed from their packages will be partially unusable.

[jacobtylerwalls]

Thanks for the extra details. I was wondering if the error was thrown only in object finalization after pylint's main program finishes. Obviously we need to clean it up either way, but this sounds more serious than the error I'd seen from time to time before.

[sivel]

The exception output is interspersed in the normal output from pylint. So it doesn't seem necessarily to happen at the very end. I'm unsure exactly the internal timing of things, it could be in object finalization, but it doesn't seem to be after the main program finishes.

[SNIP]
lib/ansible/playbook/base.py:435:12: W0201: Attribute '_ds' defined outside __init__ (attribute-defined-outside-init)
lib/ansible/playbook/base.py:716:0: C0115: Missing class docstring (missing-class-docstring)
Traceback (most recent call last):
  File "/Users/sivel/projects/ansibledev/astroid/astroid/decorators.py", line 104, in inner
    yield from generator
ValueError: generator already executing
lib/ansible/playbook/base.py:739:58: E1101: Module 'ansible.constants' has no 'ANY_ERRORS_FATAL' member (no-member)
lib/ansible/playbook/base.py:741:48: E1101: Module 'ansible.constants' has no 'TASK_TIMEOUT' member (no-member)
[SNIP]

[nitzmahone]

I mean the "easy" solution is just to override the default sys.unraisablehook behavior (since that's where it's coming from), but I've been trying to figure out why some of the decorated generators appear to be getting abandoned in the first place, and what's different in Python 3.12 that's causing them to move when they get finalized- I have a couple of ideas, but the whole thing is a bit of a Heisenbug, since the same repro doesn't manifest under a debugger, and most of the instrumentation code I'm able to add to astroid also prevents the repro from working.

Python 3.12 appears to be much more aggressive about finalizing the (apparently) abandoned wrapped generators- I was able to proxy the ones that are involved in our repro to see when they're finalized (sadly weakref also prevents our repro from working), and running that output side-by-side with 3.11, it's clear that 3.12 finalizes them much earlier than 3.11 does. I've found dozens of ways to "fix" the problem (ie "make our repro stop working"), but without understanding the root cause, I'm not at all convinced we're not just moving the problem around, and that the root cause is really another subtle bug unique to Python 3.12.

I was wondering if the error was thrown only in object finalization after pylint's main program finishes.

They aren't- it's happening inline, and much earlier with 3.12 than 3.11. I've also tried having the proxied generators capture the callstack on init, so I can examine the ones that appear to be leaked later to figure out who created them- python -X dev is helpful in deferring their finalization to exit, but unfortunately the added memory overhead also "fixes" our reliable repro. The abandoned ones do still appear though, they just don't blow up in finalization (which adds to my suspicion that this is really a subtle Python 3.12 regression of some kind, so I don't want to just propose a simple fix that masks the real problem).

[nitzmahone]

(we're also going to try a throwaway plugin that disables the exception printing in sys.unraisablehook, just to unblock our CI from using pylint 3.0.1 under 3.12, but I still think there's something wiggling under this rock that needs killing 😉 )

[nickdrozd]

I don't know if it's of any use, but here's a whittled-down version of that base.py file that reproduces the error:

from ansible.plugins.loader import action_loader

class FieldAttributeBase:
    def _load_module_defaults(self):
        resolved_action = self._resolve_action()
        if resolved_action:
            validated_defaults_dict[resolved_action] = defaults

        if True:
            resolved_action = self._resolve_action()
            if resolved_action:
                validated_defaults_dict[resolved_action] = defaults

    def _resolve_action(self):
        prefer = action_loader.find_plugin_with_context()

        if prefer.resolved:
            return prefer.resolved_fqcn

[nitzmahone]

Yeah, at least improves the signal/noise ratio while beating on it- thanks!