pip 23.1: pip install --no-index --find-links <dir> requires setuptools and wheel to be available

[mohd-akram]

Description

Unable to install packages with --no-index --find-links since release 23.1.

Expected behavior

Installation works.

pip version

23.2.1

Python version

3.11

OS

macOS Ventura

How to Reproduce

mkdir pip-test && cd pip-test
python3 -m venv venv
venv/bin/pip download --no-binary :all: leftpad
# venv/bin/pip install pip==23.0 # this makes it work
PIP_NO_CACHE_DIR=off venv/bin/pip install --no-index --find-links . leftpad

Output

Collecting leftpad
  Downloading leftpad-0.1.2.tar.gz (1.5 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Saved ./leftpad-0.1.2.tar.gz
Successfully downloaded leftpad
Looking in links: .
Processing ./leftpad-0.1.2.tar.gz
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      Looking in links: .
      ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none)
      ERROR: No matching distribution found for setuptools>=40.8.0
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

Since Pip 23.1 build isolation is default, the error indictates that setuptools is not available in the isolated build environment.

Either you can provide setuptools so it can be used on the build environment, or you can pass the the flag for no build isolation.

[mohd-akram]

Passing --no-build-isolation does not work:

Looking in links: .
Processing ./leftpad-0.1.2.tar.gz
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [11 lines of output]
      Warning: 'classifiers' should be a list, got type 'tuple'
      running dist_info
      creating /private/var/folders/mw/d2jp6b715v59p7ghfyp6rr5c0000gn/T/pip-modern-metadata-xbktabm9/leftpad.egg-info
      writing /private/var/folders/mw/d2jp6b715v59p7ghfyp6rr5c0000gn/T/pip-modern-metadata-xbktabm9/leftpad.egg-info/PKG-INFO
      writing dependency_links to /private/var/folders/mw/d2jp6b715v59p7ghfyp6rr5c0000gn/T/pip-modern-metadata-xbktabm9/leftpad.egg-info/dependency_links.txt
      writing top-level names to /private/var/folders/mw/d2jp6b715v59p7ghfyp6rr5c0000gn/T/pip-modern-metadata-xbktabm9/leftpad.egg-info/top_level.txt
      writing manifest file '/private/var/folders/mw/d2jp6b715v59p7ghfyp6rr5c0000gn/T/pip-modern-metadata-xbktabm9/leftpad.egg-info/SOURCES.txt'
      reading manifest file '/private/var/folders/mw/d2jp6b715v59p7ghfyp6rr5c0000gn/T/pip-modern-metadata-xbktabm9/leftpad.egg-info/SOURCES.txt'
      writing manifest file '/private/var/folders/mw/d2jp6b715v59p7ghfyp6rr5c0000gn/T/pip-modern-metadata-xbktabm9/leftpad.egg-info/SOURCES.txt'
      creating '/private/var/folders/mw/d2jp6b715v59p7ghfyp6rr5c0000gn/T/pip-modern-metadata-xbktabm9/leftpad-0.1.2.dist-info'
      error: invalid command 'bdist_wheel'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

It's clear that pip is capable of building the package, it just doesn't want to:

venv/bin/pip download --no-binary :all: wheel flit_core
PIP_NO_CACHE_DIR=off venv/bin/pip install --no-index --find-links . leftpad # fails
# Okay, manually install wheel first
PIP_NO_CACHE_DIR=off venv/bin/pip install --no-index --find-links . wheel
PIP_NO_CACHE_DIR=off venv/bin/pip install --no-index --find-links . leftpad # now it works

When you install wheel in this way, you get this in the log "Building wheels for collected packages: wheel". So it has wheel internally, and it has wheel in the links folder. It refuses to use either without manual prodding.

[notatallshaw]

That's a different error, passing in no build isolation did solve the error about no available setuptools.

[pradyunsg]

Please see https://pip.pypa.io/en/stable/reference/build-system/ -- that has a lot of information about how pip builds things, which seems to be information that'd be useful to you here.

Notably, your setup seems to be incapable of handling pyproject.toml-based isolated builds.

---

PIP_NO_CACHE_DIR=off venv/bin/pip install --no-index --find-links . leftpad

If you don't have wheel installed, pip will pick the pyproject.toml based build system. That's actually a detail that I'm realising that the linked documentation does not cover, although the changelog for 23.1 does mention it:

When the wheel package is not installed, pip now uses the default build backend instead of setup.py install and setup.py develop for project without pyproject.toml. (#8559)

A PR updating the documentation would be appropriate here. Beyond that, I don't think the command you've posed can work without also opting out of pyproject.toml-based builds or ensuring that . contains the build-dependencies of the project (for leftpad, that's setuptools and wheel, as documented here).

[mohd-akram]

If you don't have wheel installed, pip will pick the pyproject.toml based build system. That's actually a detail that I'm realising that the linked documentation does not cover, although the changelog for 23.1 does mention it:

Is there any scenario where you can install a package using pip without wheel available in some form? If there is no such scenario, why doesn't pip have wheel as a dependency?

[pradyunsg]

Is there any scenario where you can install a package using pip without wheel available in some form?

Yes, if a project has a wheel available. Alternatively, any project that doesn't depend on setuptools as a build-backend.

[mohd-akram]

Alternatively, any project that doesn't depend on setuptools as a build-backend.

Presumably such a project has an alternative backend, so this doesn't really solve the basic issue of installing a simple source package without additional dependencies, as was previously possible.

why doesn't pip have wheel as a dependency?

I really cannot think of any reason why this isn't the case given pip's heavy reliance on wheels, to the point that there is a wheel subcommand which presumably does not function until wheel is installed. This should have been done in 23.1 and would have avoided a lot of unnecessary breakage.

[notatallshaw]

to the point that there is a wheel subcommand which presumably does not function until wheel is installed

Reading the docs:

• https://pip.pypa.io/en/stable/cli/pip_wheel/
• https://pip.pypa.io/en/stable/reference/build-system/
• https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/
• https://peps.python.org/pep-0517/

I get the impression you don't need wheel to run pip wheel if you're using an alterntive build backend like flit.

FYI, it may help to understand the situation better by looking at it from the right perspective, specifically Pip is not a package manager it is a package installer. It does not manage a packages lifecycle or the envoronment, for that sort of thing you should look at one of PDM, Poetry, Huak, Rye, Pixi, etc.

If packages are built ahead of time as wheels then Pip needs none of these dependencies, if pip was like npm install then it simply wouldn't accept source distributions and you would be required to build ahead of time.

But as source distrubitions are historic to Python and must be supported Pip uses standard build hooks, but it can not include the entire universe of all build systems including every single version (and updating them as they are released) that any random package might require.

[mohd-akram]

I should mention the original problem that caused me to open this issue, which btw isn't resolved even when both setuptools and (an older version of) wheel are available:

venv/bin/pip download --no-binary :all: wheel==0.38.4 setuptools
PIP_NO_CACHE_DIR=off venv/bin/pip install --no-index --find-links . leftpad

Output:

LookupError: file:///private/tmp/pip-test/setuptools-68.2.0.tar.gz is already being built: setuptools>=40.8.0 from file:///private/tmp/pip-test/setuptools-68.2.0.tar.gz

Yet another reason why pip should include all its core functionality built-in, like every other package manager. Alternative backends are just that, alternative - 99.9% of the Python ecosystem uses wheel, and there's no point in pretending otherwise. Anyway, closing this issue since this is behaving as intended based on the comments here.