Allow using binary wheels instead of building from source

[real-yfprojects]

flatpak-builder version

1.0.10

Linux distribution and version

Ubuntu 20.04

Affected flatpak-builder tool

pip/flatpak-pip-generator

flatpak-builder tool cli args

No response

Source repository URL

No response

Flatpak-builder manifest URL

No response

Description

When generating a dependency file for the python package cryptography from pypi.org, the flatpak-pip-generator recognised that the package only supplies platform dependent wheel distributions and therefore includes the source distribution . However the source distribution can only be build by compiling complicated rust dependencies and building the app with flatpak-builder fails therefore. I solved this by replacing the source distribution with all the different wheel packages available so that the right one is always available. Why can't flatpak-pip-generator do something similar?

PS: Which wheel distributions do I actually need for building the flatpak on aarch64 and x86?

[TingPing]

The tool is just very conservative. Source builds, when they actually build, will always work. Wheel binaries depend on host packages which are unknown and may not work.

This has been discussed elsewhere but never done. A flag to simply say --use-wheels-for=cryptography where you can override this would be reasonable.

[johannesjh]

This would be awesome for installing packages like numpy, scikit-learn or pandas as binary wheels since these are difficult to build+install from source packages (because of their many compile time dependencies).

What would the output of flatpak-pip-generator look like? In case the wheels are platform-specific, I guess the output would have to specify multiple wheels, one for each target architecture, right?

How could this output be generated? The flatpak-pip-generator script currently calls pip to resolve packages. It could be possible to resolve packages for a specific target architecture by passing the --platform parameter to pip.

[real-yfprojects]

What would the output of flatpak-pip-generator look like? In case the wheels are platform-specific, I guess the output would have to specify multiple wheels, one for each target architecture, right?

You simply add all wheels needed to sources and specify their platform using only_arches.

[real-yfprojects]

How could this output be generated? The flatpak-pip-generator script currently calls pip to resolve packages. It could be possible to resolve packages for a specific target architecture by passing the --platform parameter to pip.

It is also easily possible to query the pypi for a list of wheels available for a given package. In fact this is already implemented in the script.

[johannesjh]

What would the output of flatpak-pip-generator look like? In case the wheels are platform-specific, I guess the output would have to specify multiple wheels, one for each target architecture, right?

You simply add all wheels needed to sources and specify their platform using only_arches.

sounds good! I'm trying to think this through and make it as specific as possible. can you provide an example of how you would structure the flatpak json? a separate module for each platform-specific wheel, right?

It is also easily possible to query the pypi for a list of wheels available for a given package. In fact this is already implemented in the script.

yes, querying pypi is the easy part... resolving the right wheel filename to install is the hard part. for this, the flatpak-pip-generator script currently relies on the pip download command to choose the right filename.

• querying is easy: for example, if I were to call flatpak-pip-generator scikit-learn, then flatpak-pip-generator's get_pypi_url method would query https://pypi.org/pypi/scikit-learn/json and fetch a lot of information. Click the link and see for yourself, the json structure lists all files for all releases.

• choosing the right filename for the "best" wheel to install is hard. For example, the json structure for scikit-learn's current 1.1.1 release includes 17 files. The filenames have PEP425 semantics, i.e., each filename specifies what python version, abi and platform it is been made for. How would we choose the "best" wheel to install amongst so many options?

How then would we solve the hard part?, i.e., how can we choose the right filename, the best wheel to install?

• Option A) we could script a heuristic to choose the "best" wheel. I liked this explanatory blogposts of PEP425 tags, and there is functionality in the python packaging package for parsing PEP425 tags. This would allow a script that filters wrong options (e.g., windows or mac_os wheels) and then implements some heuristic to choose whatever option is "best". Difficult to say what would be best, e.g., is a platform-specific wheel better than a platform-independent one, if both are offered?
• Option B) we could keep relying on the functionality of pip download to choose the "best" wheel for any given constraints. That is what flatpak-pip-generator is doing already, see

  flatpak-builder-tools/pip/flatpak-pip-generator

  Line 228 in a86a157

  pip_download = flatpak_cmd + [

  . So I guess we would have to pass --platform arguments to pip download in order to download wheels for a specific platform. And then we would have to skip the step where flatpak-pip-generator downloads source packages instead of wheels.

[real-yfprojects]

sounds good! I'm trying to think this through and make it as specific as possible. can you provide an example of how you would structure the flatpak json? a separate module for each platform-specific wheel, right?

This is how I did it. Right now pip will choose the correct wheel when running the build process.
Some of the wheels might not be needed for any build. In fact only two of them should be needed since the flatpak build server only runs on two platforms. The x-checker-data field is not used since the bot for that can't check platform dependent wheels afaik.

{
  "name": "python3-secretstorage",
  "buildsystem": "simple",
  "build-commands": [
    "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} \"secretstorage\" --no-build-isolation"
  ],
  "sources": [
    {
      "type": "file",
      "url": "https://files.pythonhosted.org/packages/79/b2/78bd6b9705296a8030c398619c9dedaa0724199be800955a7c18a1e6a3ba/scikit_learn-1.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
      "sha256": "33cf061ed0b79d647a3e4c3f6c52c412172836718a7cd4d11c1318d083300133",
      "only-arches": ["aarch64"]
    },
    {
      "type": "file",
      "url": "https://files.pythonhosted.org/packages/43/bc/7130ffd49a1cf72659c61eb94d8f037bc5502c94866f407c0219d929e758/scikit_learn-1.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
      "sha256": "47464c110eaa9ed9d1fe108cb403510878c3d3a40f110618d2a19b2190a3e35c",
      "only-arches": ["x86_64"]
    },
    {
      "type": "file",
      "url": "https://files.pythonhosted.org/packages/58/be/06987c1268a5c6beea0fea7b3c25eb52839fa23693ab2f92b80721d78554/scikit_learn-1.1.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
      "sha256": "e851f8874398dcd50d1e174e810e9331563d189356e945b3271c0e19ee6f4d6f",
      "only-arches": ["aarch64"]
    },
    {
      "type": "file",
      "url": "https://files.pythonhosted.org/packages/72/7d/cbcad2588a4baf1661e43005a9c35a955ab38e247a943715d90a7c96e6b3/scikit_learn-1.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
      "sha256": "b928869072366dc138762fe0929e7dc88413f8a469aebc6a64adc10a9226180c",
      "only-arches": ["x86_64"]
    },
    {
      "type": "file",
      "url": "https://files.pythonhosted.org/packages/21/f1/08f5e313c028bfce28abc068ba5b6633ed95b767441b6e5271249ae65601/scikit_learn-1.1.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
      "sha256": "8ff56d07b9507fbe07ca0f4e5c8f3e171f74a429f998da03e308166251316b34",
      "only-arches": ["aarch64"]
    },
    {
      "type": "file",
      "url": "https://files.pythonhosted.org/packages/62/cb/49d4c9d3505b0dd062f49c4f573995977876cc556c658caffcfcd9043ea8/scikit_learn-1.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
      "sha256": "c2dad2bfc502344b869d4a3f4aa7271b2a5f4fe41f7328f404844c51612e2c58",
      "only-arches": ["x86_64"]
    }
  ]
}

[real-yfprojects]

How then would we solve the hard part?, i.e., how can we choose the right filename, the best wheel to install?

I think Option B is the way to go since that only adds code for passing the options --platform, --python-version, --implementation and --abi. However we still need to know what to pass for these options since the script should return the (almost) same results on every machine.

[johannesjh]

This is how I did it.

great, thank you! so basically you downloaded a list of all wheels, filtered the list for the linux platform with desired architecture, and added the wheels as sources to the manifest. later on, during the build, pip install will pick whatever wheel it considers to be best from the available sources. that's clever, clean and easy, I like this solution!

[johannesjh]

However we still need to know what to pass for these options

draft:

• python interpreter: defaults to currently used python version, and that should be just fine
  EDIT: the python version that is installed in the target runtime. fallback: the system's python version.
• abi, must be any of the following values: cp3* or none
• platform: must be manylinux* or any

I continued thinking and researching about how to filter the list of wheel filenames (from pypi json), so we only include roughly suitable candidates as sources in the build manifest:

• quick and dirty filtering using regexes could be used to blacklist strings such as macos or win. should be enough to get started.
• precise filtering of matching wheels is less difficult than I would have thought. I found this code from pyodide that implements a find_matching_wheels function. It boils down to comparing two list of tags. In our case, the first list of tags would come from packaging.tags.sys_tags; this is a list of tags that are compatible with the current system, in preferential order. The second list of tags would come from parsing a wheel's filename from the pypi .json using packaging.tags.parse_tag; this is the list of tags that the wheel has been built for.