Add --cache-dir option to pip-compile

[richafrank]

Ensure tests pass without dependencies cached from previous tests.

Running tests individually, I was seeing intermittent failures depending on whether the cache was populated or not, specifically test_stdin and test_filter_pip_markers. Also I refactored a test to use an existing fixture.

Changelog-friendly one-liner: Add --cache-dir option to pip-compile.

Contributor checklist

• Provided the tests for the changes.
• Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
• Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[codecov]

Codecov Report

Merging #1022 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1022      +/-   ##
==========================================
- Coverage   99.11%   99.11%   -0.01%     
==========================================
  Files          34       34              
  Lines        2364     2360       -4     
  Branches      303      302       -1     
==========================================
- Hits         2343     2339       -4     
  Misses         11       11              
  Partials       10       10

Impacted Files	Coverage Δ
piptools/resolver.py	100% <ø> (ø)	⬆️
piptools/utils.py	100% <ø> (ø)	⬆️
piptools/cache.py	100% <100%> (ø)	⬆️
piptools/scripts/compile.py	100% <100%> (ø)	⬆️
tests/test_cli_compile.py	100% <100%> (ø)	⬆️
piptools/repositories/pypi.py	92.92% <100%> (ø)	⬆️
tests/test_top_level_editable.py	100% <100%> (ø)	⬆️
tests/test_utils.py	100% <100%> (ø)	⬆️
tests/conftest.py	100% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0da08e5...9ce14df. Read the comment docs.

[atugushev]

Hello @richafrank,

Thanks for bringing this up! I like the intention. Is there a way to isolate the DependencyCache.cache_dir explicitly for each test, rather than clear the cache? The workaround with clearing works, but if we want to run tests in parallel someday the isolation would help. The cache directory could be configured via an environment variable (e.g. PIP_TOOLS_CACHE_DIR). Also, this could be useful for #395. Please, let me know what you think.

[richafrank]

Thanks @atugushev . I can definitely make it more explicit!

Since this is manifesting in the cli tests, my recommendation would be to add a --cache-dir cli arg to pip-compile that falls back to an env var, as you suggested, followed by the current CACHE_DIR, and then change the parameter to Resolver from a cache instance to the cache dir to pass through. The only consumers of the cache parameter right now are the tests, which only configure the cache_dir anyway. For isolation in the tests then, we could have the runner fixture provide a CliRunner that uses a tmpdir for --cache-dir.

Adding a cli arg in addition to the env var would mean more transparency and give more flexibility, allowing tests to run in parallel in the same process even. But that's more to change, so curious your thoughts.

[richafrank]

(I updated this branch with my suggestion from above, if that sounds good.)

[richafrank]

Unfortunately, as long as we have a default cache directory, one could easily add a test that relies on it unintentionally. That would be a good reason to add back the autouse clear_cache fixture as well.

[atugushev]

Thank you for working on it! I'd prefer to use an auto-used fixture wich sets the PIP_TOOLS_CACHE_DIR to a temporary directory globally. This should decrease the patch size, see the details below.

[richafrank]

Thanks @atugushev . Sounds like I went a bit too explicit! Your suggestions look good to me.

[atugushev]

Great! Thanks for accepting my suggestions. Left a few minor comments below:

[richafrank]

👍 on the cleanup

[atugushev]

I've looked deeply at the PR and found that cache dir is also using in PyPIRepository class. See the details below.

[atugushev]

Awesome! 👏👏👏

[atugushev]

@richafrank thanks for the contribution!

[richafrank]

Thanks @atugushev !

[atugushev]

pip-tools v4.4.0 is released!