pip-sync uninstalls pkg-resources, breaks psycopg2

[IlyaSemenov]

pip-sync uninstalls pkg-resources, which in turn breaks installation of many other packages. pkg-resources is a new "system" package that was recently extracted from setuptools (since version 31, I believe). I think it must be handled similarly to setuptools.

Steps to replicate

On a fully updated Ubuntu 16.04 LTS:

semenov@dev2:~/tmp$ rm -rf ~/.cache/pip
semenov@dev2:~/tmp$ virtualenv --python=$(which python3) test
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/semenov/tmp/test/bin/python3
Also creating executable in /home/semenov/tmp/test/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
semenov@dev2:~/tmp$ cd test
semenov@dev2:~/tmp/test$ . bin/activate
(test) semenov@dev2:~/tmp/test$ pip install pip-tools
Collecting pip-tools
  Downloading pip_tools-1.8.0-py2.py3-none-any.whl
Collecting six (from pip-tools)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting first (from pip-tools)
  Downloading first-2.0.1-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools)
  Downloading click-6.6-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 559kB/s
Installing collected packages: six, first, click, pip-tools
Successfully installed click-6.6 first-2.0.1 pip-tools-1.8.0 six-1.10.0
(test) semenov@dev2:~/tmp/test$ echo psycopg2 > requirements.in
(test) semenov@dev2:~/tmp/test$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
psycopg2==2.6.2
(test) semenov@dev2:~/tmp/test$ pip-sync
Uninstalling pkg-resources-0.0.0:
  Successfully uninstalled pkg-resources-0.0.0
Collecting psycopg2==2.6.2
  Downloading psycopg2-2.6.2.tar.gz (376kB)
    100% |████████████████████████████████| 378kB 2.4MB/s
Could not import setuptools which is required to install from a source distribution.
Traceback (most recent call last):
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/pip/req/req_install.py", line 387, in setup_py
    import setuptools  # noqa
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/setuptools/__init__.py", line 10, in <module>
    from setuptools.extern.six.moves import filter, filterfalse, map
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in <module>
    from pkg_resources.extern import VendorImporter
ImportError: No module named 'pkg_resources.extern'

Traceback (most recent call last):
  File "/home/semenov/tmp/test/bin/pip-sync", line 11, in <module>
    sys.exit(cli())
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/piptools/scripts/sync.py", line 72, in cli
    install_flags=install_flags))
  File "/home/semenov/tmp/test/lib/python3.5/site-packages/piptools/sync.py", line 157, in sync
    check_call([pip, 'install'] + pip_flags + install_flags + sorted(to_install))
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip', 'install', 'psycopg2==2.6.2']' returned non-zero exit status 1

Expected result

pip-sync keeps pkg-resources in place, psycopg2 installs normally.

Actual result

pip-sync uninstalls pkg-resources, then psycopg2 installation fails with: ImportError: No module named 'pkg_resources.extern'

[merwok]

Debian and derivatives have split pkg-resources from setuptools for a long time. I ran into this issue in my first ten minutes of trying out pip-tools.

[rnwolf]

I have a similar issue:

`rnwolf  ~  workspace  cpptest4  pip-sync
Uninstalling pkg-resources-0.0.0:
Successfully uninstalled pkg-resources-0.0.0

rnwolf  ~  workspace  cpptest4  pip-compile > requirements.txt
Traceback (most recent call last):
File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/bin/pip-compile", line 7, in
from piptools.scripts.compile import cli
File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/lib/python3.5/site-packages/piptools/scripts/compile.py", line 16, in
from ..repositories import LocalRequirementsRepository, PyPIRepository
File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/lib/python3.5/site-packages/piptools/repositories/init.py", line 3, in
from .pypi import PyPIRepository
File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/lib/python3.5/site-packages/piptools/repositories/pypi.py", line 14, in
from ..cache import CACHE_DIR
File "/home/rnwolf/workspace/cpptest4/.direnv/python-3.5.2/lib/python3.5/site-packages/piptools/cache.py", line 9, in
from pkg_resources import Requirement
ImportError: No module named 'pkg_resources'
`

[merwok]

Work-around: pip uninstall setuptools && pip install -U setuptools

[noamraph]

A work-around that worked for me: add a line pkg-resources==0.0.0 to requirements.in.

@merwok 's workaround didn't work for me.

[jedie]

A work-around that worked for me: add a line pkg-resources==0.0.0 to requirements.in.

doesn't work for me.
pip-sync output:

Collecting pkg-resources==0.0.0
  Could not find a version that satisfies the requirement pkg-resources==0.0.0 (from versions: )
No matching distribution found for pkg-resources==0.0.0

@merwok 's workaround is ugly, but worked here.

[opokatech]

Adding pkg-resources==0.0.0 to requirements.in worked for me. Tested in following way on debian stretch:
mkvirtualenv tmp
mkdir tmp && cd tmp
echo pkg-resources==0.0.0 > requirements.in
pip install pip-tools
echo flask >> requirements.in
pip-compile
pip-sync

After that flask has been installed and pkg-resources is still there. I tested it also by adding bpython to requirements.in (and pip-compile && pip-sync) and removing it followed by pip-compile and pip-sync. It worked without any problems, i.e. pkg-resources was preserved.

[franekp]

It not only breaks psycopg2, but also pip-tools itself!