Add 'pip cache' command #6391

duckinator · 2019-04-08T18:43:01Z

This PR adds pip cache command, as per #4685.

TODO:

(Thanks to ~~@BrownTruck~~ @xavfernandez and @tdsmith for their attempts, in #3734 and #3968 respectively, which I took a lot of inspiration/ideas from.)

pradyunsg · 2019-04-09T05:27:38Z

Thanks for this PR @duckinator! :D

I think script.pip(...) is creating a subprocess, and modifies the environment when doing so.

Yep. I can confirm this.

@BrownTruck

That PR was by @xavfernandez that @BrownTruck (a bot) rebased. No worries on getting this detail wrong tho. :P

pradyunsg · 2019-04-23T19:03:16Z

Hey @duckinator!

This is still on my radar; I just haven't had much time outside of RL to get to answering your question.

duckinator · 2019-05-10T14:10:41Z

Rebased, but not made any changes.

xavfernandez · 2019-05-11T22:09:18Z

Tests are using https://github.com/pypa/pip/blob/master/tests/conftest.py#L98-L166 which should help you find the correct cache directory.

chrahunt · 2019-07-13T20:22:42Z

The basic issue appears to be that changes to APPDATA and LOCALAPPDATA in the environment of the current process do not impact the returned value of ctypes.windll.shell32.SHGetFolderPathW. Only for sub-processes.

One option could be to take the current value of os.environ['LOCALAPPDATA'] and append \pip\Cache to it, duplicating the logic from appdirs.user_cache_dir.

Another option could be to get the cache dir from a sub-process itself, like script.run('python', '-c', 'from pip._internal.locations import USER_CACHE_DIR; print(USER_CACHE_DIR). This seems like more of a hack, but does avoid the duplication.

In either case, it may be good to make it a property of PipTestEnvironment so it's accessible from the required tests or any others that might need the cache directory.

BrownTruck · 2019-07-28T20:15:04Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will be eligible for code review and hopefully merging!

duckinator · 2019-08-14T21:09:50Z

I'll try to get around to rebasing this and updating it taking into account @chrahunt's comment soon. Sorry for the delay!

duckinator · 2019-09-25T04:28:29Z

I opened issue #7082 about accessing environment variables that are modified in subprocesses.

duckinator · 2019-09-25T04:42:21Z

Okay, looks like I'm not going to be able to add a fix for the Windows tests tonight, but I rebased my changes, opened #7082, and have a rough idea of how I want to approach it.

src/pip/_internal/commands/cache.py

Co-Authored-By: Pradyun Gedam <pradyunsg@gmail.com>

…e}` behave more predictably.

…any arguments.

It bothered me that _whether or not a function had a certain argument_ dictated the contents of a directory. Pytest fixtures are inherently kinda magical, but that was a bit much for me.

…line length problems.

…ests.

… cache.

…e_matches_wheel().

…ng the path separator.

duckinator · 2020-04-01T20:56:03Z

Rebased and resolved the merge conflict. /cc @pradyunsg

sbidoul · 2020-04-01T21:08:21Z

@pradyunsg it would be great to have this one 20.1 too.

pradyunsg · 2020-04-13T14:22:58Z

We're going to have a beta period for pip 20.1, and we can collect reports about how this works during the beta. Merging since I think it's very reasonable to have users try out this fairly new and big user facing chunk to our beta period and give us feedback. :)

Thanks for working on this @duckinator! This command has been a long time coming -- your patience and work on this is much appreciated! ^>^

pradyunsg · 2020-04-13T14:23:47Z

Alsssooo, thanks everyone who reviewed this or made suggestions here! Much appreciated! ^>^

duckinator · 2020-04-13T21:43:30Z

@pradyunsg I'm excited that it's finally been merged! Do you know what the timing for releasing the 20.1 beta will be?

pradyunsg · 2020-04-13T21:46:51Z

It's April 20th - #7951 is the tracking issue. :)

118: Update pip to 20.1 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.0.2** to **20.1**. <details> <summary>Changelog</summary> ### 20.1 ``` ================= Process ------- - Document that pip 21.0 will drop support for Python 2.7. Features -------- - Add ``pip cache dir`` to show the cache directory. (`7350 <https://github.com/pypa/pip/issues/7350>`_) Bug Fixes --------- - Abort pip cache commands early when cache is disabled. (`8124 <https://github.com/pypa/pip/issues/8124>`_) - Correctly set permissions on metadata files during wheel installation, to permit non-privileged users to read from system site-packages. (`8139 <https://github.com/pypa/pip/issues/8139>`_) ``` ### 20.1b1 ``` =================== Deprecations and Removals ------------------------- - Remove emails from AUTHORS.txt to prevent usage for spamming, and only populate names in AUTHORS.txt at time of release (`5979 <https://github.com/pypa/pip/issues/5979>`_) - Remove deprecated ``--skip-requirements-regex`` option. (`7297 <https://github.com/pypa/pip/issues/7297>`_) - Building of local directories is now done in place, instead of a temporary location containing a copy of the directory tree. (`7555 <https://github.com/pypa/pip/issues/7555>`_) - Remove unused ``tests/scripts/test_all_pip.py`` test script and the ``tests/scripts`` folder. (`7680 <https://github.com/pypa/pip/issues/7680>`_) Features -------- - pip now implements PEP 610, so ``pip freeze`` has better fidelity in presence of distributions installed from Direct URL requirements. (`609 <https://github.com/pypa/pip/issues/609>`_) - Add ``pip cache`` command for inspecting/managing pip's wheel cache. (`6391 <https://github.com/pypa/pip/issues/6391>`_) - Raise error if ``--user`` and ``--target`` are used together in ``pip install`` (`7249 <https://github.com/pypa/pip/issues/7249>`_) - Significantly improve performance when ``--find-links`` points to a very large HTML page. (`7729 <https://github.com/pypa/pip/issues/7729>`_) - Indicate when wheel building is skipped, due to lack of the ``wheel`` package. (`7768 <https://github.com/pypa/pip/issues/7768>`_) - Change default behaviour to always cache responses from trusted-host source. (`7847 <https://github.com/pypa/pip/issues/7847>`_) - An alpha version of a new resolver is available via ``--unstable-feature=resolver``. (`988 <https://github.com/pypa/pip/issues/988>`_) Bug Fixes --------- - Correctly freeze a VCS editable package when it is nested inside another VCS repository. (`3988 <https://github.com/pypa/pip/issues/3988>`_) - Correctly handle ``%2F`` in URL parameters to avoid accidentally unescape them into ``/``. (`6446 <https://github.com/pypa/pip/issues/6446>`_) - Reject VCS URLs with an empty revision. (`7402 <https://github.com/pypa/pip/issues/7402>`_) - Warn when an invalid URL is passed with ``--index-url`` (`7430 <https://github.com/pypa/pip/issues/7430>`_) - Use better mechanism for handling temporary files, when recording metadata about installed files (RECORD) and the installer (INSTALLER). (`7699 <https://github.com/pypa/pip/issues/7699>`_) - Correctly detect global site-packages availability of virtual environments created by PyPA’s virtualenv>=20.0. (`7718 <https://github.com/pypa/pip/issues/7718>`_) - Remove current directory from ``sys.path`` when invoked as ``python -m pip <command>`` (`7731 <https://github.com/pypa/pip/issues/7731>`_) - Stop failing uninstallation, when trying to remove non-existent files. (`7856 <https://github.com/pypa/pip/issues/7856>`_) - Prevent an infinite recursion with ``pip wheel`` when ``$TMPDIR`` is within the source directory. (`7872 <https://github.com/pypa/pip/issues/7872>`_) - Significantly speedup ``pip list --outdated`` by parallelizing index interaction. (`7962 <https://github.com/pypa/pip/issues/7962>`_) - Improve Windows compatibility when detecting writability in folder. (`8013 <https://github.com/pypa/pip/issues/8013>`_) Vendored Libraries ------------------ - Update semi-supported debundling script to reflect that appdirs is vendored. - Add ResolveLib as a vendored dependency. - Upgrade certifi to 2020.04.05.1 - Upgrade contextlib2 to 0.6.0.post1 - Upgrade distro to 1.5.0. - Upgrade idna to 2.9. - Upgrade msgpack to 1.0.0. - Upgrade packaging to 20.3. - Upgrade pep517 to 0.8.2. - Upgrade pyparsing to 2.4.7. - Remove pytoml as a vendored dependency. - Upgrade requests to 2.23.0. - Add toml as a vendored dependency. - Upgrade urllib3 to 1.25.8. Improved Documentation ---------------------- - Emphasize that VCS URLs using git, git+git and git+http are insecure due to lack of authentication and encryption (`1983 <https://github.com/pypa/pip/issues/1983>`_) - Clarify the usage of --no-binary command. (`3191 <https://github.com/pypa/pip/issues/3191>`_) - Clarify the usage of freeze command in the example of Using pip in your program (`7008 <https://github.com/pypa/pip/issues/7008>`_) - Add a "Copyright" page. (`7767 <https://github.com/pypa/pip/issues/7767>`_) - Added example of defining multiple values for options which support them (`7803 <https://github.com/pypa/pip/issues/7803>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <github-bot@pyup.io>

duckinator mentioned this pull request Apr 8, 2019

Add a command for managing pip's cache #4685

Closed

duckinator force-pushed the pip-cache branch 3 times, most recently from 299cf3c to 5303ee5 Compare April 8, 2019 20:10

This comment has been minimized.

Sign in to view

pradyunsg added C: cache Dealing with cache and files in it type: enhancement Improvements to functionality labels Apr 22, 2019

This comment has been minimized.

Sign in to view

duckinator force-pushed the pip-cache branch from 5303ee5 to 039cfee Compare May 10, 2019 13:54

BrownTruck added the needs rebase or merge PR has conflicts with current master label Jul 28, 2019

duckinator force-pushed the pip-cache branch from 039cfee to acb159a Compare September 25, 2019 03:41

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Sep 25, 2019

duckinator force-pushed the pip-cache branch 2 times, most recently from 3eecaef to bea48bc Compare September 25, 2019 03:48

This comment has been minimized.

Sign in to view

duckinator mentioned this pull request Sep 25, 2019

Add ability to access env vars that are changed in test-related subprocesses #7082

Closed

pradyunsg reviewed Sep 27, 2019

View reviewed changes

src/pip/_internal/commands/cache.py Outdated Show resolved Hide resolved

xavfernandez reviewed Sep 27, 2019

View reviewed changes

src/pip/_internal/commands/cache.py Outdated Show resolved Hide resolved

src/pip/_internal/commands/cache.py Outdated Show resolved Hide resolved

duckinator and others added 16 commits April 1, 2020 16:54

[commands/cache] Reformat output of pip cache info

ed9f885

Co-Authored-By: Pradyun Gedam <pradyunsg@gmail.com>

[commands/cache] Fix test_cache_info test.

f8b67c8

[commands/cache] Make filenames more realistic in tests.

8b518b2

[commands/cache] Make _find_wheels(), and this `pip cache {list,remov…

d57407a

…e}` behave more predictably.

[commands/cache] Remove unnecessary re-definition of __init__.

e1fde1f

[commands/cache] Have pip cache info raise an exception if it gets …

e804aa5

…any arguments.

[tests/functional/cache] Refactor to be less redundant.

6e425d8

[tests/functional/cache] Make fixtures feel less magical.

274b295

It bothered me that _whether or not a function had a certain argument_ dictated the contents of a directory. Pytest fixtures are inherently kinda magical, but that was a bit much for me.

[tests/functional/test_cache] Always call normcase on cache dir; fix …

c6b5a52

…line length problems.

[tests/functional/cache] Rewrite all of the pip cache {list,remove} t…

32ce3ba

…ests.

[tests/functional/test_cache] Add test pip cache list with an empty…

ba7c3ac

… cache.

[tests/functional/test_cache] Split apart tests for pip cache purge.

a20b28d

[tests/functional/test_cache] Refactor list_matches_wheel() and remov…

8858237

…e_matches_wheel().

[tests/functional/test_cache] Remove unused import.

b7239f5

[tests/functional/test_cache] Fix test on Python 2.7.

0c4eafa

[tests/functional/test_cache] Use os.path.join() instead of hard-codi…

b988417

…ng the path separator.

duckinator force-pushed the pip-cache branch from 9cdb436 to b988417 Compare April 1, 2020 20:54

pradyunsg added this to the 20.1 milestone Apr 1, 2020

pradyunsg approved these changes Apr 13, 2020

View reviewed changes

pradyunsg merged commit bdff935 into pypa:master Apr 13, 2020

gutsytechster mentioned this pull request May 6, 2020

Better reporting of how the cache is utilized at runtime #2905

Open

lock bot added the auto-locked Outdated issues that have been locked by automation label May 20, 2020

lock bot locked as resolved and limited conversation to collaborators May 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add 'pip cache' command #6391

Add 'pip cache' command #6391

duckinator commented Apr 8, 2019 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pradyunsg commented Apr 9, 2019

This comment has been minimized.

pradyunsg commented Apr 23, 2019

duckinator commented May 10, 2019

xavfernandez commented May 11, 2019

chrahunt commented Jul 13, 2019 •

edited

Loading

BrownTruck commented Jul 28, 2019

duckinator commented Aug 14, 2019

This comment has been minimized.

duckinator commented Sep 25, 2019

duckinator commented Sep 25, 2019

duckinator commented Apr 1, 2020

sbidoul commented Apr 1, 2020

pradyunsg commented Apr 13, 2020

pradyunsg commented Apr 13, 2020

duckinator commented Apr 13, 2020

pradyunsg commented Apr 13, 2020 •

edited

Loading

Add 'pip cache' command #6391

Add 'pip cache' command #6391

Conversation

duckinator commented Apr 8, 2019 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pradyunsg commented Apr 9, 2019

This comment has been minimized.

pradyunsg commented Apr 23, 2019

duckinator commented May 10, 2019

xavfernandez commented May 11, 2019

chrahunt commented Jul 13, 2019 • edited Loading

BrownTruck commented Jul 28, 2019

duckinator commented Aug 14, 2019

This comment has been minimized.

duckinator commented Sep 25, 2019

duckinator commented Sep 25, 2019

duckinator commented Apr 1, 2020

sbidoul commented Apr 1, 2020

pradyunsg commented Apr 13, 2020

pradyunsg commented Apr 13, 2020

duckinator commented Apr 13, 2020

pradyunsg commented Apr 13, 2020 • edited Loading

duckinator commented Apr 8, 2019 •

edited

Loading

chrahunt commented Jul 13, 2019 •

edited

Loading

pradyunsg commented Apr 13, 2020 •

edited

Loading