pip list --outdated --format=freeze does not return the right version of the outdated packages.

[matrixise]

Description

To see the outdated packages, just pip list -o, but when I use --format=freeze, the command returns an output with the oldest versions and not the newest ones.

Expected behavior

Returns an output with the newest versions of the packages.

pip version

21.0.1

Python version

3.9.4

OS

macOS Big Sur

How to Reproduce

1. Get package from '...'
2. Then run '...'
3. An error occurs.

Output

(happiness) stephane@gandalf ~/s/g/m/happiness (main)> pip list -o
Package             Version Latest Type
------------------- ------- ------ -----
asgiref             3.3.1   3.3.4  wheel
decorator           4.4.2   5.0.5  wheel
Django              3.1.7   3.2    wheel
django-extensions   3.1.1   3.1.2  wheel
django-heroku       0.0.0   0.3.1  wheel
djangorestframework 3.12.2  3.12.4 wheel
Faker               6.5.0   8.0.0  wheel
gunicorn            20.0.4  20.1.0 sdist
idna                2.10    3.1    wheel
ipython             7.21.0  7.22.0 wheel
parso               0.8.1   0.8.2  wheel
prompt-toolkit      3.0.16  3.0.18 wheel
Pygments            2.8.0   2.8.1  wheel
urllib3             1.26.3  1.26.4 wheel
(happiness) stephane@gandalf ~/s/g/m/happiness (main)> pip list -o --format=freeze
asgiref==3.3.1
decorator==4.4.2
Django==3.1.7
django-extensions==3.1.1
django-heroku==0.0.0
djangorestframework==3.12.2
Faker==6.5.0
gunicorn==20.0.4
idna==2.10
ipython==7.21.0
parso==0.8.1
prompt-toolkit==3.0.16
Pygments==2.8.0
urllib3==1.26.3

Code of Conduct

I agree to follow the PSF Code of Conduct.

Hi all,

Here is my bug report about pip list -o --format=freeze

Thank you

[pfmoore]

If we were to put the latest versions in, that might be an incompatible set of versions. So it's not actually usable as "freeze" output.

What's your use case for this?

[matrixise]

Hi @pfmoore

Sometimes, just to update my dependencies for a PoC or MVP, I would like to use pip list -o --format=freeze | xargs -n 1 pip install -U for an update of my dependencies.

Do you have another alternative?

Have a nice day,

Stéphane

[pfmoore]

Do you have another alternative?

No, but there's a good chance that will leave you with a broken system, because you're not going to get dependencies resolved properly, so I'm not sure it's something we want to make possible...

If you don't understand why it could leave you with a broken system:

1. Individual pip install runs (the -n 1 to xargs) don't ensure that the newly installed package satisfies all the dependencies of already-installed stuff. Removing -n 1 will just make it more likely that the command will fail because of inconsistencies, and still won't ensure a fully-updated consistent system.
2. pip list --outdated lists all new versions, with no check that they are mutually consistent.

[matrixise]

yep, I know I risk breaking my system, but fortunately, I use a lot of tests for the usage of the external APIs.

About the xargs, that was just an example. Sometimes, I just update the dependencies and freeze the list in the requirements.txt file. After that, I delete/recreate the environment and restart a fresh install of the dependencies and in this case, I can see if there is an issue with the graph of the dependencies.

But we start to go outside of the main discussion ;-)

So, in this case, I see pip list -o --format=freeze does not give the same result as pip list -o and --format is just for the output formatting. is it right? maybe we could indicate it in the documentation or the --help section of the CLI. I have filled this issue just because that was not clear for me, maybe it's the case for other people.

Have a nice day,

[piotr-dobrogost]

As pip list --outdated lists two versions for each package in its output one can argue which of these two is the right one that should be listed when used with --format=freeze.
I can see how the interaction between --outdated and --format=freeze options might not be clear at first sight.
If you know the history and the fact that --outdated was added later than --format=freeze you kind of can guess that --outdated acts in this case as a filter only and does not change the meaning of --format=freeze which is to list packages and versions obtained from pip list in a specific format.
In other words when you use both --outdated and --format=freeze you can think of the latter as overriding the former so to speak.

[pfmoore]

To be clear, it's certainly possible to report the "new" version in the freeze output. It's also possible to disallow the freeze format when using --outdated, as it doesn't allow reporting "current" and "latest" versions, which is fundamental to the "outdated" display. It's also possible to do nothing (except maybe document more clearly that the freeze format shows the installed version).

The question is, whether having pip list --outdated --format=freeze report a set of requirements that looks like it upgrades everything, but which in practice has a high probability of being inconsistent, is a good UI in the first place. It feels like a bug magnet to me, and even if @matrixise knows it risks breaking his system, I bet a lot of our users wouldn't, would assume it's a great way of doing --upgrade-all, break their systems, and come here raising bugs.

Actually, if you want the output @matrixise expects, the following gives it:

> py -m pip list --outdated --format=json | jq -r '.[] | .name+\"=\"+.latest_version'
idna=3.1
setuptools=54.2.0
urllib3=1.26.4

(excuse Powershell quoting rules, for Unix probably just omit the backslashes).

[piotr-dobrogost]

It's also possible to disallow the freeze format when using --outdated, as it doesn't allow reporting "current" and "latest" versions, which is fundamental to the "outdated" display.

I think this should be done to avoid needless confusion.

[uranusjr]

I agree, let’s make the two mutually exclusive.

[nvllsvm]

--format=json is a more explicit replacement when combined with some Python glue:

pip list --outdated --format=json | python -c '
import json
import sys

for item in json.loads(sys.stdin.read()):
    print("=".join([item["name"], item["latest_version"]]))
'
}

[ashr123]

maybe I've missed it, but how can I upgrade safely all my packages on my system? (linux user)

[pfmoore]

maybe I've missed it, but how can I upgrade safely all my packages on my system?

It's not straightforward. The best way is probably to use a virtual environment and simply recreate it. Or if you're using your system-managed environment, your system package manager should handle it (assuming you've not installed packages to your system environment using pip). Otherwise, "manually, and carefully" is the best answer I can give...

Just to be clear, pip list --outdated --format=freeze wasn't a safe way to upgrade all the packages on your system, either, which is why we removed it (it gave the impression it was safe when it wasn't).

[rautamiekka]

A prime way to destroy an SSD and kill data-capped Internet connections, not to mention a massive waste of time, instead of providing a sensible means, namely --upgrade-all.

Instead the SysAdmin must learn complicated scripting (jq is seriously complicated, PowerShell not exactly easy either), when pip could internally do this much easier with existing functions.

0 steps forward. 2 or 3 steps backwards.

[pfmoore]

--upgrade-all has been on our list of useful improvements for ages now, since the new resolver was rolled out. PRs are, of course, welcome 🙂

[ashr123]

@pfmoore is there an open issue for that? If so can you link it?

[pradyunsg]

#4551

I'm gonna lock this, because I'm pretty sure there's not much more new coming out of a this discussion, that isn't already mentioned there. If you have something new to add, please do so on the linked issue.