Could pip install mirror the recommended installation command?

[zed]

The docs recommend on Mac/Linux:

$ python setup.py fetch --all build --enable-all-extensions install

Is it possible that pip install apsw would install the equivalent version and (optionally) provide a binary install where possible (OSX, Windows)? (I know, pip install does something different now).

If somebody doesn't like the recommended options; they may use the explicit python setup.py command with their preferred options.

[rogerbinns]

Unfortunately that isn't possible. pip just runs setup.py without allowing the user to specify any additional options. setup.py couldn't practically tell the difference between pip invoking it and a user or other script invoking it.

APSW is not available on PyPI/pip because of their limitations. I've not seen any progress on that front. (The underlying reason is their sweet spot is distributing python code, and they are poorer at distributing C code and binaries.) A random person has put an older version of APSW on PyPI but it won't work for the vast majority of people.

[zed]

It s irrelevant whether pip install accepts any arguments or not. Obviously, setup.py should use the recommended options by default -- no need to pass any options.

[rogerbinns]

It isn't practical for setup.py to use the recommended options by default, otherwise I would have long ago! setup.py is mostly a data structure written in python, and it is distutils (or setuptools if present) that decides what to do. In particular note that there a number of Commands, and distutils/setuptools devices which ones to run in which order, combining their own internal logic as well as command line options, other files (eg setup.cfg), and whatever their caller stated if there is one (eg pip).

https://docs.python.org/2/distutils/setupscript.html

[gnarlyquack]

I have gotten apsw to install via pip using the following incantation:

pip install <url> --global-option=fetch --global-option=--all --global-option=build --global-option=--enable-all-extensions

--global-option, apparently, just passes the arguments along to the underlying script. Currently I have a script that uses the GitHub API to programmatically determine the url of the latest release. It's hella ugly, but it works.

[rogerbinns]

@karlnack that is very helpful. Do you know how long pip has had that option? I'll have to do a bit more investigation as it looks like global options can only be inserted before the build command. (For example it would be a good idea if test could be run which has to come after the build command.)

[gnarlyquack]

No, I don't. In full disclosure, I actually swiped that command from another project here on GitHub, and it seemed to work. Offhand, I don't remember which though.

[zed]

@rogerbinns I see that setup.py already uses custom fetch, build, test, etc commands i.e., the default for the options is under your control already e.g., what happens if you set build_enable_all_extensions=True in setup.py or self.all=True in fetch.initialize_options()? I understand there could be backward-compatibility issues, new options to override the new default might be needed, and setup.py is already complicated as it is.

@karlnack pip install + --build-options / --global-options / --install-options is better than python setup.py (the uninstall should work) but it forces a potentially unnecessary compilation (and all the associated hurdles). It is desirable that pip install would pull a verified up-to-date binary wheel appropriate for the platform from PyPI (where it is possible).

[rogerbinns]

@zed the bit you are missing is that distutils/setuptools is what calls the various commands. I have no control over which ones are executed or in which order. build_enable_all_extensions is an ugly hack I use because there are two build commands under the hood - build and build_ext. An end user only invokes build, while distutils/setuptools also invokes build_ext. The global variable is used to pass the --enable-all-extensions setting from build to build_ext.

Setting self.all=True in fetch won't do anything useful. The fetch command is only run if distutils/setuptools decides to run it, which they do by requiring it being specified on the command line or potentially elsewhere. Making the default True would just make it as though --all had been specified, providing fetch runs in the first place.

I am going to try and make a pip install with the global options command line, and if it works add that to the doc. It will require compilation. Windows users do already have prepackaged builds, although they can't be installed with pip (not that I have tried ...).

[zed]

If fetch is not invoked automatically; include the necessary files into the release tarball.

What stops your custom build* command from building all extensions when it is called (something will call it if the installation is from sources)?

The prepackaged binaries do not correspond to the recommended command (they do not contain all extensions).

As far as I understand, if you upload binary wheels such as these but that support all extensions to PyPI when pip install apsw will use them:

pip can install from either Source Distributions (sdist) or Wheels, but if both are present on PyPI, pip will prefer a compatible wheel.

[rogerbinns]

The prepackaged binaries do not correspond to the recommended command (they do not contain all extensions).

Which ones are missing?

As for the rest of the comment, things are more complicated, and also have changed over time. For example the fetch of SQLite gets the zip file if the target is Windows, otherwise the tar.gz. (They do have different contents.) Extensions are sometimes included with the amalgamation, and sometimes in separately downloaded files. setup.py knows about that. The --enable-all-extensions setting also knows what fetch got, so it can enable the extensions that exist. Same thing with the test code.

You should also understand just how time consuming making a release is. The non-Windows side only takes a minute or two. Windows takes several hours, and is a lot more painful. I'm not too keen on making even more work for myself :-)

[zed]

Which ones are missing?

The ICU extension.

I understand that my suggestions do not contain all the details that might be needed to implement the feature. I would need to submit a pull request, to provide all the details (what is the source code after all: it is a way for one programmer to communicate to another programmer what the computer should do with enough detail).

I believe it is possible to change the defaults to correspond to the recommended options without complicating the release process more than it is now. I may be wrong.

[rogerbinns]

I don't want to discourage you, and there is definitely no need for pull requests etc. The general problem is that all this stuff is complicated, there are many tradeoffs, some things are done for historical reasons (but should be re-examined) etc. Changing things also requires updated test code and documentation. They often are far more effort than the change.

I'be made #218 for the Windows/ICU issue. You are welcome to convince me otherwise there :-)