Concurrent upgrade of setuptools and other sdist packages fails

[jaraco]

Beginning with Setuptools 34 (pypa/setuptools#581), there's a bit of a race condition when setuptools is being upgraded as part of a larger install operation. Users have reported here's what happens:

1. Pip encounters the requirement to upgrade setuptools, downloads the wheel.
2. Pip downloads sdists for other packages to be built and installed.
3. Pip uninstalls setuptools.
4. Pip attempts to build other package but setuptools isn't yet fully installed (its dependencies are missing).

Here's a build where the issue occurs.

I believe when #4144 lands, pip can create an implied dependency of setuptools on every package to provide compatibility with the prior expectation.

In the interim, packages and environments affected should pin to an old version of setuptools or (preferably) upgrade setuptools in a separate step prior to upgrading other packages. See python-excel/xlrd#194 for an example.

[cromulus]

I think it's relevant to note that the error results in a broken installation of pip3:

$pip3
Traceback (most recent call last):
  File "/usr/bin/pip3", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 70, in <module>
    import packaging.version
ImportError: No module named 'packaging'

[jaraco]

@cromulus: The issue you're seeing there is unrelated to the issue in the OP. The issue you have is a broken install of setuptools and pkg_resources. You can probably repair with python3 -m pip install setuptools.

Edit: on second thought, perhaps you're right that the failed pip operation does leave the environment in a broken state.

[jaraco]

What I wonder is why is it that all of the uninstalls/installs done in a single, atomic operation. In other words, why aren't the builds all done before setuptools gets uninstalled?

[jaraco]

Aha. This issue was addressed in #2478 and #2616. Using pip 6.1 or later avoids the condition altogether.

[dstufft]

Oh, so is this a non-issue then if you're using a modern version of pip?

[jaraco]

Yes, that's right. The topological sorting means that setuptools' dependencies are installed before setuptools is, so even when setuptools is upgraded, it's in a viable state.