diff --git a/peps/pep-0751.rst b/peps/pep-0751.rst index a60811048b6..a2d090ad63d 100644 --- a/peps/pep-0751.rst +++ b/peps/pep-0751.rst @@ -31,9 +31,9 @@ Currently, no standard exists to: - Create an immutable record, such as a lock file, of which dependencies were installed. -Considering there are at least four well-known solutions to this problem in the -community (``pip freeze``, pip-tools_, Poetry_, and PDM_), there seems to be an -appetite for lock files in general. +Considering there are at least five well-known solutions to this problem in the +community (``pip freeze``, pip-tools_, uv_, Poetry_, and PDM_), there seems to +be an appetite for lock files in general. Those tools also vary in what locking scenarios they support. For instance, ``pip freeze`` and pip-tools only generate lock files for the current @@ -63,11 +63,11 @@ used to create their lock file. The file format is designed to be human-readable. This is so that the contents of the file can be audited by a human to make sure no -undesired dependencies end up being included in the lock file. It is also to -facilitate easy understanding of what would be installed if the lock file -without necessitating running a tool, once again to help with auditing. Finally, -the format is designed so that viewing a diff of the file is easy by centralizing -relevant details. +undesired dependencies end up being included in the lock file. It is also +designed to facilitate easy understanding of what would be installed from the +lock file without necessitating running a tool, once again to help with +auditing. Finally, the format is designed so that viewing a diff of the file is +easy by centralizing relevant details. The file format is also designed to not require a resolver at install time. Being able to analyze dependencies in isolation from one another when listed in @@ -108,6 +108,11 @@ This is similar to what ``pip freeze`` and pip-tools_ support, but with more strictness of the exact files as well as incorporating support to specify the locked files for multiple environments in the same file. +Per-file locking should be used when the installation attempt should fail +outright if there is no explicitly pre-approved set of installation artifacts +for the target platform. For example: locking the deployment dependencies for a +managed web service. + Package Locking =============== @@ -133,6 +138,12 @@ what environments their contributors are working from. As already mentioned, this approach is supported by PDM_. Poetry_ has `shown some interest `__. +Per-package locking should be used when the exact set of potential target +platforms is not known when generating the lock file, as it allows installation +tools to choose the most appropriate artifacts for each platform from the +pre-approved set. For example: locking the development dependencies for an open +source project. + ============= Specification @@ -182,7 +193,7 @@ other is disallowed. - String - The name of the hash algorithm used for calculating all hash values. - Only a single hash algorithm is used for the entire file to allow the - ``[[package.files]]`` table to be written inline for readability and + ``[[packages.files]]`` table to be written inline for readability and compactness purposes by only listing a single hash value instead of multiple values based on multiple hash algorithms. - Specifying a single hash algorithm guarantees that an algorithm that the user @@ -205,30 +216,30 @@ other is disallowed. representing the direct, top-level dependencies to be installed. -``[[file-lock]]`` -================= +``[[file-locks]]`` +================== - Array of tables - Mutually exclusive with ``[package-lock]``. - The array's existence implies the use of the per-file locking approach. - An environment that meets all of the specified criteria in the table will be considered compatible with the environment that was locked for. -- Lockers MUST NOT generate multiple ``[file-lock]`` tables which would be +- Lockers MUST NOT generate multiple ``[file-locks]`` tables which would be considered compatible for the same environment. - In instances where there would be a conflict but the lock is still desired, either separate lock files can be written or per-package locking can be used. -- Entries in array SHOULD be sorted by ``file-lock.name`` lexicographically. +- Entries in array SHOULD be sorted by ``file-locks.name`` lexicographically. -``file-lock.name`` ------------------- +``file-locks.name`` +------------------- - String - A unique name within the array for the environment this table represents. -``[file-lock.marker-values]`` ------------------------------ +``[file-locks.marker-values]`` +------------------------------ - Optional - Table of strings @@ -240,17 +251,18 @@ other is disallowed. updating the file. -``file-lock.wheel-tags`` ------------------------- +``file-locks.wheel-tags`` +------------------------- - Optional - Array of strings -- An unordered array of `wheel tags`_ which must be supported by the environment. +- An unordered array of `wheel tags`_ for which all tags must be supported by + the environment. - The array MAY not be exhaustive to allow for a smaller array as well as to - help prevent multiple ``[[file-lock]]`` tables being compatible with the + help prevent multiple ``[[file-locks]]`` tables being compatible with the same environment by having one array being a strict subset of another - ``file-lock.wheel-tags`` entry in the same file's - ``[[file-lock]]`` tables. + ``file-locks.wheel-tags`` entry in the same file's + ``[[file-locks]]`` tables. - Lockers SHOULD sort the keys lexicographically to minimize changes when updating the file. - Lockers MUST NOT include @@ -263,7 +275,7 @@ other is disallowed. ================== - Table -- Mutually exclusive with ``[[file-lock]]``. +- Mutually exclusive with ``[[file-locks]]``. - Signifies the use of the package locking approach. @@ -278,49 +290,49 @@ other is disallowed. same information. -``[[package]]`` -=============== +``[[packages]]`` +================ - Array of tables - The array contains all data on the locked package versions. -- Lockers SHOULD record packages in order by ``package.name`` lexicographically - and ``package.version`` by the sort order for `version specifiers`_. +- Lockers SHOULD record packages in order by ``packages.name`` lexicographically + and ``packages.version`` by the sort order for `version specifiers`_. - Lockers SHOULD record keys in the same order as written in this PEP to - minimmize changes when updating. + minimize changes when updating. - Designed so that relevant details as to why a package is included are in one place to make diff reading easier. -``package.name`` ----------------- +``packages.name`` +----------------- - String -- The `normalized name`_ of the package. +- The `normalized name`_ of the packages. - Part of what's required to uniquely identify this entry. -``package.version`` -------------------- +``packages.version`` +-------------------- - String -- The version of the package. +- The version of the packages. - Part of what's required to uniquely identify this entry. -``package.multiple-entries`` ----------------------------- +``packages.multiple-entries`` +----------------------------- - Boolean - If package locking via ``[package-lock]``, then the multiple entries for the - same package MUST be mutually exclusive via ``package.marker`` (this is not - required for per-file locking as the ``package.*.lock`` entries imply mutual + same package MUST be mutually exclusive via ``packages.marker`` (this is not + required for per-file locking as the ``packages.*.lock`` entries imply mutual exclusivity). - Aids in auditing by knowing that there are multiple entries for the same package that may need to be considered. -``package.description`` ------------------------ +``packages.description`` +------------------------ - Optional - String @@ -329,21 +341,21 @@ other is disallowed. purpose. -``package.simple-repo-package-url`` ------------------------------------ +``packages.simple-repo-package-url`` +------------------------------------ - Optional (although mutually exclusive with - ``package.files.simple-repo-package-url``) + ``packages.files.simple-repo-package-url``) - String - Stores the `project detail`_ URL from the `Simple Repository API`_. - Useful for generating Packaging URLs (aka PURLs). - When possible, lockers SHOULD include this or - ``package.files.simple-repo-package-url`` to assist with generating + ``packages.files.simple-repo-package-url`` to assist with generating `software bill of materials`_ (aka SBOMs). -``package.marker`` ------------------- +``packages.marker`` +------------------- - Optional - String @@ -354,8 +366,8 @@ other is disallowed. installed. -``package.requires-python`` ---------------------------- +``packages.requires-python`` +---------------------------- - Optional - String @@ -366,11 +378,11 @@ other is disallowed. ``package-lock.requires-python`` was chosen. - It should not provide useful information for installers as it would be captured by ``package-lock.requires-python`` and isn't relevant when - ``[[file-lock]]`` is used. + ``[[file-locks]]`` is used. -``package.dependents`` ----------------------- +``packages.dependents`` +----------------------- - Optional - Array of strings @@ -380,8 +392,8 @@ other is disallowed. - This does not provide information which influences installers. -``package.dependencies`` ------------------------- +``packages.dependencies`` +------------------------- - Optional - Array of strings @@ -389,62 +401,62 @@ other is disallowed. - Useful in analyzing why a package happens to be listed in the file for auditing purposes. - This does not provide information which influences the installer as - ``[[file-lock]]`` specifies the exact files to use and ``[package-lock]`` - applicability is determined by ``package.marker``. + ``[[file-locks]]`` specifies the exact files to use and ``[package-lock]`` + applicability is determined by ``packages.marker``. -``package.direct`` ------------------- +``packages.direct`` +------------------- - Optional (defaults to ``false``) - Boolean - Represents whether the installation is via a `direct URL reference`_. -``[[package.files]]`` ---------------------- +``[[packages.files]]`` +---------------------- -- Must be specified if ``[package.vcs]`` is not +- Must be specified if ``[packages.vcs]`` is not - Array of tables - Tables can be written inline. - Represents the files to potentially install for the package and version. -- Entries in ``[[package.files]]`` SHOULD be lexicographically sorted by - ``package.files.name`` key to minimze changes in diffs. +- Entries in ``[[packages.files]]`` SHOULD be lexicographically sorted by + ``packages.files.name`` key to minimze changes in diffs. -``package.files.name`` -'''''''''''''''''''''' +``packages.files.name`` +''''''''''''''''''''''' - String - The file name. - Necessary for installers to decide what to install when using package locking. -``package.files.lock`` -'''''''''''''''''''''' +``packages.files.lock`` +''''''''''''''''''''''' -- Required when ``[[file-lock]]`` is used +- Required when ``[[file-locks]]`` is used - Array of strings -- An array of ``file-lock.name`` values which signify that the file is to be - installed when the corresponding ``[[file-lock]]`` table applies to the +- An array of ``file-locks.name`` values which signify that the file is to be + installed when the corresponding ``[[file-locks]]`` table applies to the environment. -- There MUST only be a single file with any one ``file-lock.name`` entry per +- There MUST only be a single file with any one ``file-locks.name`` entry per package, regardless of version. -``package.files.simple-repo-package-url`` -''''''''''''''''''''''''''''''''''''''''' +``packages.files.simple-repo-package-url`` +'''''''''''''''''''''''''''''''''''''''''' - Optional (although mutually exclusive with - ``package.simple-repo-package-url``) + ``packages.simple-repo-package-url``) - String -- The value has the same meaning as ``package.simple-repo-package-url``. +- The value has the same meaning as ``packages.simple-repo-package-url``. - This key is available per-file to support :pep:`708` when some files override what's provided by another `Simple Repository API`_ index. -``package.files.origin`` -'''''''''''''''''''''''' +``packages.files.origin`` +''''''''''''''''''''''''' - Optional - String @@ -453,8 +465,8 @@ other is disallowed. for the file if not already downloaded/available. -``package.files.hash`` -'''''''''''''''''''''' +``packages.files.hash`` +''''''''''''''''''''''' - String - The hash value of the file contents using the hash algorithm specified by @@ -463,17 +475,17 @@ other is disallowed. with. -``[package.vcs]`` ------------------ +``[packages.vcs]`` +------------------ -- Must be specified if ``[[package.files]]`` is not (although may be specified - simultaneously with ``[[package.files]]``). +- Must be specified if ``[[packages.files]]`` is not (although may be specified + simultaneously with ``[[packages.files]]``). - Table representing the version control system containing the package and version. -``package.vcs.type`` -'''''''''''''''''''' +``packages.vcs.type`` +''''''''''''''''''''' - String - The type of version control system used. @@ -482,15 +494,15 @@ other is disallowed. of the direct URL data structure. -``package.vcs.origin`` -'''''''''''''''''''''' +``packages.vcs.origin`` +''''''''''''''''''''''' - String - The URI of where the repository was located when the lock file was generated. -``package.vcs.commit`` -'''''''''''''''''''''' +``packages.vcs.commit`` +''''''''''''''''''''''' - String - The commit ID for the repository which represents the package and version. @@ -498,53 +510,53 @@ other is disallowed. (e.g. no Git tags). -``package.vcs.lock`` -'''''''''''''''''''' +``packages.vcs.lock`` +''''''''''''''''''''' -- Required when ``[[file-lock]]`` is used +- Required when ``[[file-locks]]`` is used - An array of strings -- An array of ``file-lock.name`` values which signify that the repository at the - specified commit is to be installed when the corresponding ``[[file-lock]]`` +- An array of ``file-locks.name`` values which signify that the repository at the + specified commit is to be installed when the corresponding ``[[file-locks]]`` table applies to the environment. - A name in the array may only appear if no file listed in - ``package.files.lock`` contains the name for the same package, regardless of + ``packages.files.lock`` contains the name for the same package, regardless of version. -``package.directory`` ---------------------- +``packages.directory`` +---------------------- - Optional and only valid when ``[package-lock]`` is specified - String - A local directory where a source tree for the package and version exists. -- Not valid under ``[[file-lock]]`` as this PEP does not make an attempt to +- Not valid under ``[[file-locks]]`` as this PEP does not make an attempt to specify a mechanism for verifying file contents have not changed since locking was performed. -``[[package.build-requires]]`` ------------------------------- +``[[packages.build-requires]]`` +------------------------------- - Optional -- An array of tables whose structure matches that of ``[[package]]``. +- An array of tables whose structure matches that of ``[[packages]]``. - Each entry represents a package and version to use when building the enclosing package and version. -- The array is complete/locked like ``[[package]]`` itself (i.e. installers - follow the same installation procedure for ``[[package.build-requires]]`` as - ``[[package]]``) +- The array is complete/locked like ``[[packages]]`` itself (i.e. installers + follow the same installation procedure for ``[[packages.build-requires]]`` as + ``[[packages]]``) - Selection of which entries to use for an environment as the same as - ``[[package]]`` itself, albeit only applying when installing the build + ``[[packages]]`` itself, albeit only applying when installing the build back-end and its dependencies. - This helps with reproducibility of the building of a package by recording either what was or would have been used if the locker needed to build the - package. + packages. - If the installer and user choose to install from source and this array is missing then the installer MAY choose to resolve what to install for building at install time, otherwise the installer MUST raise an error. -``[package.tool]`` ------------------- +``[packages.tool]`` +------------------- - Optional - Table @@ -571,7 +583,7 @@ Expectations for Lockers - When creating a lock file for ``[package-lock]``, the locker SHOULD read the metadata of **all** files that end up being listed in - ``[[package.files]]`` to make sure all potential metadata cases are covered + ``[[packages.files]]`` to make sure all potential metadata cases are covered - If a locker chooses not to check every file for its metadata, the tool MUST either provide the user with the option to have all files checked (whether that is opt-in or out is left up to the tool), or the user is somehow notified @@ -580,9 +592,9 @@ Expectations for Lockers - Lockers MAY want to provide a way to let users provide the information necessary to install for multiple environments at once when doing per-file locking, e.g. supporting a JSON file format which specifies wheel tags and - marker values much like in ``[[file-lock]]`` for which multiple files can be + marker values much like in ``[[file-locks]]`` for which multiple files can be specified, which could then be directly recorded in the corresponding - ``[[file-lock]]`` table (if it allowed for unambiguous per-file locking + ``[[file-locks]]`` table (if it allowed for unambiguous per-file locking environment selection) .. code-block:: JSON @@ -598,13 +610,13 @@ Expectations for Installers --------------------------- - Installers MAY support installation of non-binary files - (i.e. source distributions, source trees, and VCS), but are not required to + (i.e. source distributions, source trees, and VCS), but are not required to. - Installers MUST provide a way to avoid non-binary file installation for - reproducibility and security purposes + reproducibility and security purposes. - Installers SHOULD make it opt-in to use non-binary file installation to - facilitate a secure-by-default approach + facilitate a secure-by-default approach. - Under per-file locking, if what to install is ambiguous then the installer - MUST raise an error + MUST raise an error. Installing for per-file locking @@ -612,27 +624,29 @@ Installing for per-file locking An example workflow is: -- Iterate through each ``[[file-lock]]`` table to find the one that applies to - the environment being installed for -- If no compatible environment is found an error MUST be raised -- If multiple environments are found to be compatible then an error MUST be raised -- For the compatible environment, iterate through each entry in ``[[package]]`` -- For each ``[[package]]`` entry, iterate through ``[[package.files]]`` to look - for any files with ``file-lock.name`` listed in ``package.files.lock`` +- Iterate through each ``[[file-locks]]`` table to find the one that applies to + the environment being installed for. +- If no compatible environment is found an error MUST be raised. +- If multiple environments are found to be compatible then an error MUST be + raised. +- For the compatible environment, iterate through each entry in + ``[[packages]]``. +- For each ``[[packages]]`` entry, iterate through ``[[packages.files]]`` to + look for any files with ``file-locks.name`` listed in ``packages.files.lock``. - If a file is found with a matching lock name, add it to the list of candidate - files to install and move on to the next ``[[package]]`` entry -- If no file is found then check if ``package.vcs.lock`` contains a match (no - match is also acceptable) -- If a ``[[package.files]]`` contains multiple matching entries an error MUST - be raised due to ambiguity for what is to be installed -- If multiple ``[[package]]`` entries for the same package have matching files - an error MUST be raised due to ambiguity for what is to be installed + files to install and move on to the next ``[[packages]]`` entry. +- If no file is found then check if ``packages.vcs.lock`` contains a match (no + match is also acceptable). +- If a ``[[packages.files]]`` contains multiple matching entries an error MUST + be raised due to ambiguity for what is to be installed. +- If multiple ``[[packages]]`` entries for the same package have matching files + an error MUST be raised due to ambiguity for what is to be installed. - Find and verify the candidate files and/or VCS entries based on their hash or - commit ID as appropriate + commit ID as appropriate. - If a source distribution or VCS was selected and - ``[[package.build-requires]]`` exists, then repeat the above process as - appropriate to install the build dependencies necessary to build the package -- Install the candidate files + ``[[packages.build-requires]]`` exists, then repeat the above process as + appropriate to install the build dependencies necessary to build the package. +- Install the candidate files. Installing for package locking @@ -641,22 +655,23 @@ Installing for package locking An example workflow is: - Verify that the environment is compatible with - ``package-lock.requires-python``; if it isn't an error MUST be raised -- Iterate through each entry in ``[package]]`` -- For each entry, if there's a ``package.marker`` key, evaluate the expression - - - If the expression is false, then move on - - Otherwise the package entry must be installed somehow -- Iterate through the files listed in ``[[package.files]]``, looking for the - "best" file to install -- If no file is found, check for ``[package.vcs]`` -- If no match is found, an error MUST be raised + ``package-lock.requires-python``; if it isn't an error MUST be raised. +- Iterate through each entry in ``[packages]]``. +- For each entry, if there's a ``packages.marker`` key, evaluate the expression. + + - If the expression is false, then move on. + - Otherwise the package entry must be installed somehow. + +- Iterate through the files listed in ``[[packages.files]]``, looking for the + "best" file to install. +- If no file is found, check for ``[packages.vcs]``. +- If no match is found, an error MUST be raised. - Find and verify the selected files and/or VCS entries based on their hash or - commit ID as appropriate + commit ID as appropriate. - If the match is a source distribution or VCS and - ``[[package.build-requires]]`` is provided, repeat the above as appropriate to - build the package -- Install the selected files + ``[[packages.build-requires]]`` is provided, repeat the above as appropriate + to build the package. +- Install the selected files. ======================= @@ -690,7 +705,7 @@ within the file in a ``[tool]`` entry or via a side channel external to the lock file itself. This PEP does not do anything to prevent a user from installing an incorrect -package. While including many details to help in auditing a package's inclusion, +packages. While including many details to help in auditing a package's inclusion, there isn't any mechanism to stop e.g. name confusion attacks via typosquatting. Lockers may be able to provide some UX to help with this (e.g. by providing download counts for a package). @@ -759,7 +774,7 @@ At one point, to handle the issue of metadata varying between files and thus require examining every released file for a package and version for accurate locking results, the idea was floated to introduce a new core metadata version which would require all metadata for all wheel files be the same for a single -version of a package. Ultimately, though, it was deemed unnecessary as this PEP +version of a packages. Ultimately, though, it was deemed unnecessary as this PEP will put pressure on people to make files consistent for performance reasons or to make indexes provide all the metadata separate from the wheel files themselves. As well, there's no easy enforcement mechanism, and so community @@ -938,5 +953,6 @@ CC0-1.0-Universal license, whichever is more permissive. .. _Simple Repository API: https://packaging.python.org/en/latest/specifications/simple-repository-api/ .. _software bill of materials: https://www.cisa.gov/sbom .. _TOML: https://toml.io/ +.. _uv: https://github.com/astral-sh/uv .. _version specifiers: https://packaging.python.org/en/latest/specifications/version-specifiers/ .. _wheel tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/