Rebuilding the Wheel

This repo is a prototype of completely re-building a dependency tree of Python wheels from source.

The idea is to explore what is involved with providing a Python package index from which a user can pip install knowing:

1. The binary package they are installing was built from source in a known build environment compatible with their own environment
2. All of the package’s dependencies were also built from source - any binary package installed will have been built from source
3. All of the build tools used to build these binary packages will also have been built from source

The mirror-sdists script does the following:

• Creates an empty package repository for wheels
• Downloads all source distributions under sdists-repo/downloads/ using pip download and generates a PEP503 “simple” package repository using pypi-mirror
• Three types of dependencies are also downloaded:
  • Firstly, any build system dependency specified in the pyproject.toml build-system.requires section as per PEP517
  • Secondly, any build backend dependency returned from the get_requires_for_build_wheel() build backend hook (PEP517 again)
  • Lastly, any install-time dependencies of the project as per the wheel’s core metadata Requires-Dist list.
• These dependencies are downloaded recursively and we record the order they will need to be built bottom-up in a build-order.json file.
• Uses pip wheel to build a binary package, only downloading dependencies from the fresh wheel repository
• Places the newly built wheel in the package repository and regenerates the repository listing

Finally, the install-from-mirror script installs the dependency stack into a virtual environment from this newly created repository of wheels.

Using the indexes

Fedora CPU-only builds

The https://pyai.fedorainfracloud.org/experimental/cpu index includes builds without GPU-specific optimizations. Use the https://pyai.fedorainfracloud.org/experimental/cpu/+simple/ index with pip to install packages from the index.

$ python3 -m venv numpy-test
$ source numpy-test/bin/activate
$ pip install --index-url https://pyai.fedorainfracloud.org/experimental/cpu/+simple/ numpy

RHEL 9 CUDA builds

The https://pyai.fedorainfracloud.org/experimental/cuda-ubi9 index includes builds with support for CUDA-based GPUs from Nvidia that are compatible with RHEL 9. Use the https://pyai.fedorainfracloud.org/experimental/cuda-ubi9/+simple/ index with pip to install packages from the index.

$ python3 -m venv numpy-test
$ source numpy-test/bin/activate
$ pip install --index-url https://pyai.fedorainfracloud.org/experimental/cuda-ubi9/+simple/ numpy

Additional docs

• How Tos
• Some packages have customizations applied
• Customization instructions
• Developer Instructions