Virtualenv.do_install does not work for various Python packages

[adamtheturtle]

What you were trying to do (and why)

I am a maintainer of an application which ships a Homebrew recipe so that users can easily install a CLI on macOS.
The application is written in Python 3.
The application has multiple dependencies.

The recipe has the following layout, but with many more resources:

minidcos.rb:

class Minidcos < Formula
  include Language::Python::Virtualenv

  url "https://codeload.github.com/dcos/dcos-e2e/legacy.tar.gz/2019.05.24.1"
  homepage "http://minidcos.readthedocs.io/en/latest/"
  depends_on "python3"
  depends_on "pkg-config"

  resource "httplib2" do
    url "https://files.pythonhosted.org/packages/67/33/29779c5aaeac796679a37bf798b3c2adbfaae7dbf13e966b0ab9c3aa06c0/httplib2-0.12.3.tar.gz"
    sha256 "a18121c7c72a56689efbf1aef990139ad940fee1e64c6f2458831736cd593600"
  end

  def install
    virtualenv_install_with_resources
  end

end

Some of those resources specify a custom build-backend in their [build-system] section in their pyproject.toml.

There is a pip bug which prevents these packages from installing with --no-binary :all:.

Homebrew's Virtualenv.do_install uses pip's --no-binary :all: option:

brew/Library/Homebrew/language/python.rb

Lines 260 to 264 in 6b2dbbc

	def do_install(targets)
	targets = [targets] unless targets.is_a? Array
	@formula.system @venv_root/"bin/pip", "install",
	"-v", "--no-deps", "--no-binary", ":all:",
	"--ignore-installed", *targets

It is not clear to me why --no-binary :all: is used in Homebrew, so I don't want to just say "Homebrew would be better without this option". Perhaps this is a philosophical decision.

However, links on that pip issue show that folks are using workarounds to avoid using --no-binary all.

My hope is that either this option can be removed, or there can be an easy option for recipe maintainers to avoid --no-binary :all:.

What happened (include command output)

Command output

~/D/foobar ~> brew install minidcos.rb

==> Downloading https://codeload.github.com/dcos/dcos-e2e/legacy.tar.gz/2019.05.24.1

Already downloaded: /Users/adam/Library/Caches/Homebrew/downloads/4b8bccbb4ca6db124ea6a6abdfe0968853065133307c5fe8bce33d4112f9c168--dcos-dcos-e2e-2019.05.24.1-0-gfeb3f88.tar.gz

Warning: Cannot verify integrity of 4b8bccbb4ca6db124ea6a6abdfe0968853065133307c5fe8bce33d4112f9c168--dcos-dcos-e2e-2019.05.24.1-0-gfeb3f88.tar.gz

A checksum was not provided for this resource

For your reference the SHA-256 is: f30be86f3b773d60ebe7bf8b772df31872238c24f4c84e8f5e8944810ccd0caf

==> Downloading https://files.pythonhosted.org/packages/53/c0/c7819f0bb2cf83e1b4b0d96c901b85191f598a7b534d297c2ef6dc80e2d3/virtualenv-16.6.0.tar.gz

Already downloaded: /Users/adam/Library/Caches/Homebrew/downloads/6d18e6ad1a4c8e31dfc412cb5513a676119ead57a6ad5c830a0ec57ac08fd763--virtualenv-16.6.0.tar.gz

==> python3 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/minidcos--homebrew-virtualenv-20190601-55148-1189zyg/target --install-scripts=/private/tmp/minidcos--homebrew-virtualenv-20

==> python3 -s /private/tmp/minidcos--homebrew-virtualenv-20190601-55148-1189zyg/target/bin/virtualenv -p python3 /usr/local/Cellar/minidcos/2019.05.24.1/libexec

==> Downloading https://files.pythonhosted.org/packages/1f/87/9ea76ab4cdf1fd36710d9688ec36a0053067c47e753b32272f952ff206c5/flit-1.3.tar.gz

Already downloaded: /Users/adam/Library/Caches/Homebrew/downloads/588d227f6a17845ac2092aedddac49e5349fb8390ee22121f57910130a52a475--flit-1.3.tar.gz

==> /usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/pip install -v --no-deps --no-binary :all: --ignore-installed /private/tmp/minidcos--flit-20190601-55148-1g5um1n/flit-1.3

Last 15 lines from /Users/adam/Library/Logs/Homebrew/minidcos/03.pip:

Removed build tracker '/private/tmp/pip-req-tracker-laxe52t6'

ERROR: Command "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/python3.7 -u -c 'import setuptools, tokenize;file='"'"'/private/tmp/pip-req-build-id_d4tsl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-lf57luvq/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/../include/site/python3.7/flit" failed with error code 1 in /private/tmp/pip-req-build-id_d4tsl/

Exception information:

Traceback (most recent call last):

File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 178, in main

status = self.run(options, args)

File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 414, in run

use_user_site=options.use_user_site,

File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/req/init.py", line 58, in install_given_reqs

**kwargs

File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 951, in install

spinner=spinner,

File "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 776, in call_subprocess

% (command_desc, proc.returncode, cwd))

pip._internal.exceptions.InstallationError: Command "/usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/python3.7 -u -c 'import setuptools, tokenize;file='"'"'/private/tmp/pip-req-build-id_d4tsl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/tmp/pip-record-lf57luvq/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/Cellar/minidcos/2019.05.24.1/libexec/bin/../include/site/python3.7/flit" failed with error code 1 in /private/tmp/pip-req-build-id_d4tsl/
Do not report this issue to Homebrew/brew or Homebrew/core!
  

What you expected to happen

No error.

Step-by-step reproduction instructions (by running brew commands)

Create the following file minidcos.rb.
The resources are the output of poet flit from homebrew-pypi-poet.

class Minidcos < Formula
  include Language::Python::Virtualenv

  url "https://codeload.github.com/dcos/dcos-e2e/legacy.tar.gz/2019.05.24.1"
  homepage "http://minidcos.readthedocs.io/en/latest/"
  depends_on "python3"
  depends_on "pkg-config"

  resource "flit" do
    url "https://files.pythonhosted.org/packages/1f/87/9ea76ab4cdf1fd36710d9688ec36a0053067c47e753b32272f952ff206c5/flit-1.3.tar.gz"
    sha256 "6f6f0fb83c51ffa3a150fa41b5ac118df9ea4a87c2c06dff4ebf9adbe7b52b36"
  end

  def install
    virtualenv_install_with_resources
  end

end

Run brew install minidcos.rb.

brew configuration

~ ~> brew config
HOMEBREW_VERSION: 2.1.1
ORIGIN: https://github.com/Homebrew/brew
HEAD: b4f73e61649fcfc5aaa779c311e2514619ce01e7
Last commit: 7 weeks ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 98167e409139cf6fed8c289600ba079473bba6b6
Core tap last commit: 5 weeks ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_LOGS: /Users/adam/Library/Logs/Homebrew
CPU: quad-core 64-bit skylake
Homebrew Ruby: 2.3.7 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby
Clang: 10.0 build 1001
Git: 2.20.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 7.54.0 => /usr/bin/curl
macOS: 10.14.5-x86_64
CLT: 10.2.1.0.1.1554506761
Xcode: 10.2.1
CLT headers: 10.2.1.0.1.1554506761
~ ~> brew doctor
Your system is ready to brew.

[close-issue-app]

From the issue template:

Please note that we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

If you add the necessary information to this issue (don't create a new one, please) then this issue may be reopened.