Manage versioning of Poetry tool dependency #604
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some templates utilize Python packages.
The version of Python used by these templates is controlled via the `pyproject.toml` metadata file. Previously the file
specified that any version >=3.9.0 and <4.0.0 could be used, since the templates had been validated with 3.9 and
theoretically no breaking changes should be introduced without a major version bump.
However, there was a breaking change in Python version 3.10, which caused an incompatibility with the version of the
`pyyaml` Python package dependency. This caused the `poetry install` command to fail when a version of Python greater
than 3.9.x was used:
```
PEP517 build of a dependency failed
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
| Command '['/tmp/tmpu92tgtph/.venv/bin/python', '/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/pyproject_hooks/_in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmpt2syl665']' returned non-zero exit status 1.
|
| /tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
| !!
|
| ********************************************************************************
| Please consider removing the following classifiers in favor of a SPDX license expression:
|
| License :: OSI Approved :: MIT License
|
| See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
| ********************************************************************************
|
| !!
| self._finalize_license_expression()
| running egg_info
| writing lib3/PyYAML.egg-info/PKG-INFO
| writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
| writing top-level names to lib3/PyYAML.egg-info/top_level.txt
| Traceback (most recent call last):
| File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
| main()
| ~~~~^^
| File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
| json_out["return_val"] = hook(**hook_input["kwargs"])
| ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/hostedtoolcache/Python/3.13.4/x64/lib/python3.13/site-packages/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
| return hook(config_settings)
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/build_meta.py", line 331, in get_requires_for_build_wheel
| return self._get_build_requires(config_settings, requirements=[])
| ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires
| self.run_setup()
| ~~~~~~~~~~~~~~^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/build_meta.py", line 317, in run_setup
| exec(code, locals())
| ~~~~^^^^^^^^^^^^^^^^
| File "<string>", line 271, in <module>
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/__init__.py", line 115, in setup
| return distutils.core.setup(**attrs)
| ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 186, in setup
| return run_commands(dist)
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
| dist.run_commands()
| ~~~~~~~~~~~~~~~~~^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
| self.run_command(cmd)
| ~~~~~~~~~~~~~~~~^^^^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/dist.py", line 1102, in run_command
| super().run_command(command)
| ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
| cmd_obj.run()
| ~~~~~~~~~~~^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/command/egg_info.py", line 312, in run
| self.find_sources()
| ~~~~~~~~~~~~~~~~~^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/command/egg_info.py", line 320, in find_sources
| mm.run()
| ~~~~~~^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/command/egg_info.py", line 543, in run
| self.add_defaults()
| ~~~~~~~~~~~~~~~~~^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/command/egg_info.py", line 581, in add_defaults
| sdist.add_defaults(self)
| ~~~~~~~~~~~~~~~~~~^^^^^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/command/sdist.py", line 109, in add_defaults
| super().add_defaults()
| ~~~~~~~~~~~~~~~~~~~~^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/_distutils/command/sdist.py", line 245, in add_defaults
| self._add_defaults_ext()
| ~~~~~~~~~~~~~~~~~~~~~~^^
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/_distutils/command/sdist.py", line 330, in _add_defaults_ext
| self.filelist.extend(build_ext.get_source_files())
| ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
| File "<string>", line 201, in get_source_files
| File "/tmp/tmpu92tgtph/.venv/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 131, in __getattr__
| raise AttributeError(attr)
| AttributeError: cython_sources
Note: This error originates from the build backend, and is likely not a problem with poetry but one of the following issues with pyyaml (5.4.1)
- not supporting PEP 517 builds
- not specifying PEP 517 build requirements correctly
- the build requirements are incompatible with your operating system or Python version
- the build requirements are missing system dependencies (eg: compilers, libraries, headers).
You can verify this by running pip wheel --no-cache-dir --use-pep517 "pyyaml (==5.4.1)".
exit status 1
```
Although the incompatibility has already been fixed in `pyyaml`, this is a transitive dependency and so it might be
difficult to update the version used by the project. In addition, it indicates that the Python project is not maintained
in a manner respectful of the Semver specification, and so we can not trust that they will bump the major version when
making breaking changes.
So the chosen solution is to adjust the Python version constraints to require a version less than 3.10.0 be used. This
will still allow use of any Python version in the 3.9.x series to make the infrastructure more friendly to local use by
contributors, while improving the stability of the project infrastructure (hopefully the Python maintainers will at
least bump the minor version when making breaking changes).
The project Python package dependencies are managed by the "Poetry" tool. Previously, the version of Poetry was not managed in any way. The GitHub Actions workflows used whichever version of Poetry happened to be installed on the runner machine. This meant that the GitHub Actions workflows could break at any time through the Poetry installation on the runner machine being updated to an incompatible version. The contributors used whichever version of Poetry happened to be installed on their machine. This meant that they might get different results from that produced by the environment of the GitHub Actions workflows. The better solution is to take the same approach for managing the Poetry dependency as is done for the project's other dependencies: * Install a specific version of Poetry according to a single source of versioning data. * Use the Dependabot service to get automated update pull requests. The logical place to define the Poetry package dependency version is in pyproject.toml, as is done for all other Python package dependencies. Dependabot has support for two different forms of dependency data in the pyproject.toml file: * Original Poetry data format, under the `tool.poetry` table * PEP 621 format data, under the `project` table Since Poetry can't be used to manage itself (it is instead installed using the "pipx" tool), the obvious approach would be to define the Poetry dependency via the `project` table in the file. However, this is not possible because if a `tool.poetry` table is present in pyproject.toml, Dependabot ignores the dependencies data from the `project` table. So it is necessary to place the data for the Poetry dependency under the `tool.poetry` table of the file. A special dependencies group is created for this purpose. That group is configured as "optional" so that it won't be installed redundantly by `poetry install` commands. Unfortunately pipx doesn't support using pyproject.toml as a dependency configuration file so it is necessary to get the Poetry version constraint for use in the dependency argument of the `pipx install` command by parsing the project.toml file.
per1234
added
type: enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The project Python package dependencies are managed by the Poetry tool.
Previously, the version of Poetry was not managed in any way.
The GitHub Actions workflows used whichever version of Poetry happened to be installed on the runner machine. This meant that the GitHub Actions workflows could break at any time through the Poetry installation on the runner machine being updated to an incompatible version.
The contributors used whichever version of Poetry happened to be installed on their machine. This meant that they might get different results from that produced by the environment of the GitHub Actions workflows.
The better solution is to take the same approach for managing the Poetry dependency as is done for the project's other dependencies:
The logical place to define the
poetrypackage dependency version is inpyproject.toml, as is done for all other Python package dependencies.Dependabot has support for two different forms of dependency data in the
pyproject.tomlfile:tool.poetrytableprojecttableSince Poetry can't be used to manage itself (it is instead installed using the pipx tool), the obvious approach would be to define the
poetrydependency via theprojecttable in the file. However, this is not possible because if atool.poetrytable is present inpyproject.toml, Dependabot ignores the dependencies data from theprojecttable (dependabot/dependabot-core#12220). So it is necessary to place the data for thepoetrydependency under thetool.poetrytable of the file. A special dependencies group is created for this purpose. That group is configured as "optional" so that it won't be installed redundantly bypoetry installcommands.Unfortunately pipx doesn't support using
pyproject.tomlas a dependency configuration file so it is necessary to get thepoetryversion constraint for use in the dependency argument of thepipx installcommand by parsing theproject.tomlfile.Related