--sync removes pkg-resources, leading to pillow error

[ClaudiaSchulz]

• Poetry version: 1.2.1
• Python version: 3.8.10
• OS version and name: Ubuntu 20.04.3 LTS
• pyproject.toml:

[tool.poetry]
name = "test1"
version = "0.1.0"
description = ""
authors = ["Claudia"]

[tool.poetry.dependencies]
python = "~3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Issue

When running poetry install --sync, poetry removes anything pre-installed (as it should), unfortunately this includes pkg-resources (see #2673 for a discussion), which is pre-installed.
Removing this package leads to an error when trying to then poetry add pillow, as discussed in #5104.

I also tried this with Poetry 1.2.0b2, mentioned in #5104 - same problem.

[neersighted]

There's not enough detail here to reproduce this (where is a preinstalled pkg_resources coming from?) -- here's what I tried:

docker run --rm -i --entrypoint /bin/bash python:3.8 <<EOF
curl https://install.python-poetry.org | POETRY_HOME=/opt/poetry python3 - --version 1.2.1
export PATH="/opt/poetry/bin:$PATH"
poetry new foobar
pushd foobar
poetry lock
poetry install --sync
poetry add pillow
EOF

[dimbleby]

https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463 maybe

[neersighted]

Tried again with Ubuntu 20.04:

docker run --rm -i --entrypoint /bin/bash ubuntu:20.04 <<EOF
apt-get update
apt-get install -y python3 python3-venv python3-pip curl
curl https://install.python-poetry.org | POETRY_HOME=/opt/poetry python3 - --version 1.2.1
export PATH="/opt/poetry/bin:$PATH"
poetry new foobar
pushd foobar
poetry lock
poetry install --sync
poetry add pillow
EOF

[neersighted]

I would check you're really on the Poetry version you think you're on -- the ~3.8 specifier in the example pyproject.toml looks a bit suspicious since we write ^X.Y by default these days.

[dimbleby]

$ python3 -m venv ~/.virtualenvs/foo
$ ls ~/.virtualenvs/foo/lib/python3.8/site-packages/
__pycache__  easy_install.py  pip  pip-20.0.2.dist-info  pkg_resources  pkg_resources-0.0.0.dist-info  setuptools  setuptools-44.0.0.dist-info

that pkg_resources presumably.

this is with ubuntu 20.04 and the system python. If I use a deadsnakes python3.9, then it's not there:

$ ls ~/.virtualenvs/foo/lib/python3.9/site-packages/
_distutils_hack  distutils-precedence.pth  pip  pip-22.0.4.dist-info  pkg_resources  setuptools  setuptools-58.1.0.dist-info

I don't intend to dig further, my money is on it being connected to or the same as that ubuntu bug I linked yesterday

[neersighted]

Interesting -- that does look like Ubuntu is doing some extremely sketchy stuff and it's not Poetry's fault...

[dimbleby]

I wondered why poetry didn't also uninstall pip and setuptools in that case, the answer is that there's some special-case code.

So if you wanted to treat pkg-resources as a special case to cope with this, I guess that's where to do it.

[neersighted]

Right, but uninstalling that nonsense pkg_resources shouldn't break pip... I guess what might be happening is the distro-provided pip copied from the distro's site-packages is debundled and depends on that pkg_resources... If the pip was upgraded then this wouldn't be an issue (you'd get a properly vendored version from PyPI), but since it isn't, we break the pip in the virtualenv?

This really seems like a Ubuntu issue to me, in that case. This would be solved both by #6458 or Poetry using its own installer code (#6205).

[dimbleby]

it shouldn't only be about pip and installation though.

If a project is using pkg_resources and is in an environment where pkg_resources is provided by this strange debundled version - then uninstalling that package will break the project.

[neersighted]

I was assuming that if the project used pkg_resources, it would declare it, Poetry would upgrade the nonsense version to the locked version, and pip would probably not break -- but fair point, there could be more sharp edges here.

[dimbleby]

but you wouldn't declare an explicit dependency on pkg_resources, you'd declare a dependency on setuptools.

So then poetry would see no reason to keep the unbundled pkg_resources around

[neersighted]

That is a fair point -- maybe we just hold our nose and whitelist pkg_resources with a link back here explaining why...

[ClaudiaSchulz]

@neersighted that sounds like a good solution, even though I agree that the pkg_resources is somewhat sketchy (unfortunately I have no control over my container environment, so there's no way for me to use a different OS version).
But it sounds like there wouldn't be any harm in whitelisting pkg-resources in the same way that setuptools is whitelisted.

[ClaudiaSchulz]

Also, adding pgk_resources as an explicit dependency doesn't work since Poetry is unable to find the package.

[espdev]

Poetry 1.2.2 removes setuptools from my project:

$ poetry install

Installing dependencies from lock file

Package operations: 0 installs, 0 updates, 1 removal

  • Removing setuptools (65.5.0)

And google APIs do not work without pkg_resources.
I have added setuptools as dependency to the project:

$ poetry add setuptools 

It seems it works but looks very strange.

[neersighted]

This is presumably because setuptools is required by an optional dependency you are not installing, so Poetry considers it a managed package. You do explicitly depend on pkg_resources which is shipped with setuptools, so it is correct to declare a dependency.

It's hard to provide more insight without seeing your pyproject.toml -- I suggest asking in Discord or on Discussions if you have more questions.

[macgeneral]

I run into this problem for Docker images based on Ubuntu 20.04 LTS as well.

I know it's an ugly workaround, but running

python3 -m pip install --no-cache-dir --force-reinstall --upgrade setuptools

after poetry [..] --sync [..] seems to fix the otherwise broken virtual environment.

[Secrus]

In recent versions of Python setuptools (and pkg_resources) are no longer being installed in environments by default. Poetry also changed it's behaviour to match that. If you need one of those dependencies for your project, list them in requirements. If your dependency requires one of those libraries, report the issue to project maintainers. There is nothing Poetry is going to do about it anymore.