Passing --no-cache-dir should not affect whether to use a wheel

[nehaljwani]

At the moment the help output for --no-cache-dir just says: Disable the cache.

Till a week ago, I was under the impression that it meant that pip wouldn't use a cached wheel, and wouldn't cache a wheel fetched from PyPI.

What I didn't know was that when this option is enabled and a package has to be installed from an sdist, pip doesn't actually create a wheel and ends up creating egg-info directories, instead of dist-info ones.

Quick example:

$ pip install https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz
Collecting https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz
  Downloading https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz (318kB)
    100% |████████████████████████████████| 327kB 17.3MB/s 
Building wheels for collected packages: pytz <------------------------- Builds a 'wheel'
  Running setup.py bdist_wheel for pytz ... done
  Stored in directory: /home/nwani/.cache/pip/wheels/0d/29/29/5c4384ddcad4ea97cb91f7ff4ef13ab3cff23ddaf35ce02d21
Successfully built pytz
Installing collected packages: pytz
Successfully installed pytz-2018.5
$ find $PREFIX -type d | grep pytz-2018.5
/tmp/wani.1535901395/dev/lib/python2.7/site-packages/pytz-2018.5.dist-info  <------ 'dist-info'

$ pip install https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz --no-cache-dir
Collecting https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz
  Downloading https://files.pythonhosted.org/packages/ca/a9/62f96decb1e309d6300ebe7eee9acfd7bccaeedd693794437005b9067b44/pytz-2018.5.tar.gz (318kB)
    100% |████████████████████████████████| 327kB 52.1MB/s 
Installing collected packages: pytz
  Running setup.py install for pytz ... done  <------------------------ No 'wheel' ?
Successfully installed pytz-2018.5
$ find $PREFIX -type d | grep pytz-2018.5
/tmp/wani.1535901395/dev/lib/python2.7/site-packages/pytz-2018.5-py2.7.egg-info <----- 'egg-info'

In the case of '--no-cache-dir', would it not be possible for pip to still build the wheel but throw it away after installation?

[ihnorton]

In my case, this breaks installation of a library via pip install, because the wheel step includes building of a necessary shared library (which otherwise works fine using setup.py bdist_wheel and then pointing pip at the result).

[cjerdonek]

In the case of '--no-cache-dir', would it not be possible for pip to still build the wheel but throw it away after installation?

I thought it was already supposed to do this? Can you retry with the latest version of pip (19.0.3)? If it's not building a wheel, it's possible there's a separate reason for that (pip has separate logic to decide whether to build a wheel).

[ihnorton]

I was using 19.0.3 (also tried other versions). See

pip/src/pip/_internal/commands/install.py

Lines 330 to 336 in 88bcb26

	# We don't build wheels for legacy requirements if we
	# don't have wheel installed or we don't have a cache dir
	try:
	import wheel # noqa: F401
	build_legacy = bool(options.cache_dir)
	except ImportError:
	build_legacy = False

[nehaljwani]

19.0.3 also doesn't create the wheel.

[cjerdonek]

See also this code which does what you suggest, but under a different scenario:

pip/src/pip/_internal/wheel.py

Lines 783 to 786 in 88bcb26

	# Otherwise, build the wheel just for this run using the ephemeral
	# cache since we are either in the case of e.g. a local directory, or
	# no cache directory is available to use.
	return True

[ihnorton]

AFAICT that is only applicable to the "pep 517" path. In my case -- and I suspect the repro in the original issue -- pep517_requirements is empty, and the needed package is in legacy_requirements.

[cjerdonek]

Yes. Let’s keep this issue open to discuss whether the behavior should be changed.

[cjerdonek]

I see your confused emoji. What I'm saying is that instead of documenting the current behavior which is what you did in the PR #6284 that you submitted, let's instead discuss whether pip can be changed in the way that the person opening the issue first suggested:

In the case of '--no-cache-dir', would it not be possible for pip to still build the wheel but throw it away after installation?

Also, to clarify the above, the code I referenced is applicable in both the pep 517 and non-pep 517 cases in some scenarios (e.g. without --no-cache-dir). You can see that the return value of should_use_ephemeral_cache () is used in WheelBuilder.build(), prior to determining whether _build_one_pep517() or _build_one_legacy() should be used. This is also consistent with a wheel being built for the package when --no-cache-dir wasn't provided.

[ihnorton]

Changing the behavior would be great. The current behavior seems contrary to the pip install documentation which states that pip runs bdist_wheel when installing from source. That guarantee is important to me, and probably others, because bdist_wheel is familiar and at least somewhat debuggable (whereas I don't really know how to debug pip build failures because the intermediates are removed even with --no-clean).

[cjerdonek]

For historical purposes, I found where this logic was first added: #2780

I think the “ephemeral cache” was added a lot more recently, so they didn’t have as many options to address their issue back then.

[ihnorton]

I was hoping that specifying a specific (empty) --cache-dir would help. That does force a rebuild, but appears to skip bdist_wheel in the same way. Whereas deleting the global cache directory and re-installing works.

[cjerdonek]

FYI, I just filed PR #6303 to address this.

Also, I just marked this issue as an enhancement. It's possible one could argue it's a bug, but since it's debatable IMO I chose the more forgiving option.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.