Formally drop support for distutils-only projects?

[pfmoore]

The guide already assumes that people will use setuptools when building projects, but it falls short (as far as I can see) of stating that projects should not use "pure" distutils without setuptools. Newer packaging standards (for example, PEP 517) no longer support the use of distutils without setuptools.

Should the packaging guide make a formal statement that use of distutils alone is no longer supported? Or is there somewhere else that such a statement could be made?

See giampaolo/psutil#1565 for some background.

[pradyunsg]

Sounds like a good idea to me. Does anyone have any suggestions on what would be a good place to put this note?

[illume]

Formally drop support for distutils-only projects?

• makes it harder for people to create packages.
• existing working packages don't use setuptools
• there's no pep that says distutils packages should now be broken

What is the benefit here for breaking things and making it harder for people?

You said:

Newer packaging standards (for example, PEP 517) no longer support the use of distutils without setuptools.

However this is objectively not true.

pep 517 says:

If you want to use distutils, great;

[pfmoore]

What is the benefit here for breaking things and making it harder for people?

It's not so much breaking things, as acknowledging that they are already broken and in effect give packaging tools permission to not support distutils-only projects. The tools don't actually need such permission (they can make their own support choices) but in much the same way that Python's own support lifecycle makes it easier for tools to drop support for Python 2.6, a statement in the packaging guide acts as a baseline for tools to refer to.

existing working packages don't use setuptools

Behind the scenes, they actually do - pip injects setuptools into them (on the "legacy" setup.py-only code path in pip) so that packages that use distutils are actually using setuptools.

Using pip to install a package without having setuptools available will fail - even if the package claims to work with just distutils.

pep 517 says: "If you want to use distutils, great;"

I suspect that the intention of that comment was actually "if you want to use setuptools" - a lot of times people fail to make the distinction. But even if not, then I'm fine with that if someone wants to write a PEP 517 backend for distutils. It may be harder than you;'d think to do so without using setuptools (and bdist_wheel, which uses setuptools' plugin features to register as a 3rd party command), but PEP 517 would certainly allow it.

To be absolutely clear, though, I would not advocate changing any code here - that's why I raised this as a documentation issue. Rather, all I'm suggesting is that we make it clear that the packaging community won't be putting effort into keeping distutils-without-setuptools working as new standards are developed. Tools can do what they like, in reality, but we have to consider standards - and what this statement is intended to convey is that we won't force packaging standards to limit themselves to "what distutils can achieve" (PEP 517 is a good example here, the build_wheel hook is required, even though no-one has confirmed that a distutils backend could implement such a thing).

[pradyunsg]

• makes it harder for people to create packages.
• existing working packages don't use setuptools

No?

setuptools is an already-existing mechanism for keeping backward compatibility with distutils.

If you can pip install it today, you'd still be able to do so, since setuptools builds on top of (and patches) distutils to do the right thing. If you're using distutils in your setup.py, pip install xyz would use the setuptools mechanisms, since setuptools essentially replaces distutil's mechanisms with it's own when imported -- and pip makes sure to do this.

This approach is not broken -- it's been the default (and only) option for many years now and we're not changing that for many reasons that I don't want to go into. Okay, just 1 -- a "pure" distutils python setup.py install installs a package in a manner that it can't be uninstalled, a python setup.py install done using setuptools has all the required information to uninstall it.

If you want to use distutils, great;

Sure. Sadly, distutils doesn't have a PEP 517 backend, which is the way we're moving forward. Eventually, we will remove legacy code path for installations, from pip, so the only way to invoke distutils would be via setuptools.

there's no pep that says distutils packages should now be broken

They won't "be broken". We're merely documenting status quo -- No one should be directly using distutils in new projects and should move away from it in older ones. More often that not, it's a s/distutils/setuptools/.

When we "break" compatibility, is something for the setuptools maintainers to decide (it'll likely not happen anytime soon, but I'm not a setuptools maintainer).

[pradyunsg]

I'm fine with that if someone wants to write a PEP 517 backend for distutils.

Seconded.

[illume]

The honest answer would be to call setuptools the recommended option, but note that packages requiring the furthest reach should not depend on setuptools.

The linked issue lists various bugs in using setuptools. Especially when shipping code to older(and current!) distros with random versions of setuptools installed (or none at all). Calling distutils unsupported doesn't magically fix these problems with using setuptools.

• Python has a deprecation policy, and distutils has not been deprecated as of python 3.8.
• There are fine and valid reasons for not requiring or using setuptools. One of those reasons being a package that actually installs at all in some places.
• pip, and pypi are not the only tools in town. Or even the most popular ways for sharing python code.
• distros of python do update distutils in some instances.
• anyone speaking for "the" community without asking communities should really reconsider.

Since distutils is in fact supported, and not deprecated - calling it unsupported in the guide would be incorrect, misleading and all together unhelpful. At this point in time.

[pradyunsg]

distutils won't be updated to support newer systems and standards

^ this is the main problem tbh. For this reason alone, I'd want to strongly suggest users to not use distutils. Anyway, I'm gonna bow out of this thread now. I don't see this headed in a very positive direction for me.

[ncoghlan]

"from distutils.core import setup" is definitely deprecated, and we should say so in the documentation. The reason for this is that installing such projects with './setup.py install' will result in a broken Python environment, where a package is missing the metadata required to uninstall or upgrade it. If you want './setup.py install' or './setup.py bdist_wheel' to work correctly, you must use an explicit "from setuptools import setup".

However, we should also point out that for backwards compatibility, installers are required to support projects using the legacy distutils API by injecting setuptools into the execution environment, so that even though they look like they're using plain distutils, they're actually using setuptools. This is why the default PEP 517 dependencies are setuptools and wheel.

If it fits (maybe as a footnote?), we should also point out that the long term plan is to remove distutils from the regular standard library and have the distutils API be exported directly by setuptools instead: pypa/packaging-problems#127

[ncoghlan]

As far as where to document this goes, I would suggest putting a Sphinx note at the end of https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py along the lines of:

You may see some existing projects or other tutorials that import distutils.core rather than setuptools. This is a legacy approach that installers still support for backwards compatibility purposes, but using that API means that commands like setup.py bdist_wheel won't work.

And then in a footnote we could add:

Some legacy Python environments may not have setuptools installed, and may still be requiring users to install packages by running setup.py install commands, rather than providing an installer like pip. These environments will not be able to use many published packages until they environment is updated to provide a modern Python package installation client (e.g. by running python -m ensurepip)

[ncoghlan]

The distro bugs are exactly that: distro bugs.

The answer is to use a virtual environment, and avoid their broken versions.

[ncoghlan]

And yes, @illume, you should really reconsider how many people you really represent: the vast majority of Python users gave up on trying to support Linux distro Python installations a long time ago, and the distro vendors are fine with that (their view is that the distro Python is for the distro, and people shouldn't be running arbitrary third party projects in there).

So I don't know what community you believe you're speaking on behalf of, but it isn't the Python community, and it isn't the Linux distro community either.

[ncoghlan]

When we make the PR, we can link the "legacy approach" text to https://docs.python.org/3/library/distutils.html, since that page explicitly covers the fact that using distutils directly is not a recommended approach (and the reasons why).

[pfmoore]

However, we should also point out that for backwards compatibility, installers are required to support projects using the legacy distutils API by injecting setuptools into the execution environment, so that even though they look like they're using plain distutils, they're actually using setuptools. This is why the default PEP 517 dependencies are setuptools and wheel.

This isn't entirely accurate. Pip does not (and will not, in the long term) inject setuptools into projects which use PEP 517 processing (whether that's by explicit opt-in, or via the default setuptools backend). That only happens in the legacy "direct install" code path. So projects that want to use distutils under PEP 517 will need to use a distutils backend. Once the legacy code path is phased out of pip, that will be the only way to use distutils.

[ncoghlan]

Wait, there's a plan to remove the legacy code with direct setup.py invocation? That would render large parts of PyPI completely uninstallable (since nobody can go back and change old releases), so I didn't think there was any plan to completely drop that path - it would just never support the newer features like isolated builds and proper build time dependency resolution (which aren't things that stdlib only projects need).

[ncoghlan]

You also can't write a PEP 517 backend for distutils, as it doesn't generate the right metadata - instead, migrating a distutils-only project to PEP 517 necessarily means migrating it to setuptools as well.

[pfmoore]

there's a plan to remove the legacy code with direct setup.py invocation

There's no plan as such, but it's certainly something we'd look at doing at some point (and personally, I'd advocate for it). A pre-requisite for doing it would be that the equivalent new codepath (using the default setuptools backend to build a wheel and install that) worked for if not all, then at least nearly all old-style projects. But I don't see any reason why we wouldn't expect to be able to do that. But the one place that won't happen (and that's the point of this issue) is where the project explicitly uses the old distutils API rather than the recommended setuptools API.

In the long term (and again, there's no plan, so don't infer any timescales here) I think we have to accept that a volunteer project like pip can't be expected to remain compatible with arbitrarily-old code. Telling users who want to install a project that hasn't been updated in 10 years that they have to use an older version of pip doesn't seem an unreasonable position to take. It's not as if it's that hard to fork an open source project for the sole purpose of updating its setup.py to use setuptools in place of distutils.

It's not like we're a big company like Microsoft, who invest significant paid developer resource into providing that sort of backward compatibility, and even then get criticism for some of the choices that forces them to make.

You also can't write a PEP 517 backend for distutils

Of course you can. It would use setuptools behind the scenes and monkeypatch the distutils API to use setuptools, just as pip does now. It wouldn't let people install in isolated environments where they refuse to put a copy of setuptools on their local index, but that's a situation that I (personally, not speaking as a pip maintainer) see no need to support.

I'm not sure there's any point to a distutils backend in this sense - anyone who can add a pyproject.toml to specify such a backend, could just as easily update their project to use setuptools - but that's a consequence of the decision (in PEP 517) to default to the setuptools backend.

(Actually, if we really wanted to continue supporting distutils-only projects, the setuptools backend could be updated to monkeypatch the distutils API, the way pip does at the moment. The monkeypatching pip does is essentially done in the legacy equivalent of a backend code path, after all).

Maybe what you're saying is that we shouldn't drop support for distutils-only packages, because you feel that maintaining support for old releases and projects that haven't been updated in a long time on PyPI is essential. If that's your position then I'm fine with that, but with my pip maintainer hat on, I'd simply say that under PEP 517, it's the backend's responsibility to provide that support, not the frontend's, and that at some point (to be agreed) pip would stop supporting the legacy codepath and at that point backend support would need to have been sorted out (either in the setuptools backend, or by updating PEP 517 to make something else the default).

[ncoghlan]

I think I actually just misunderstood what you were suggesting, since there's two potential levels of removal here:

• removing the option to bypass PEP 517 entirely (I think we all assume that will happen at some point)
• making the default PEP 517 backend something other than setuptools.build_meta.__legacy__ (while I haven't actually tried it, I believe that backend already handles projects that import their setup function from distutils.core)

The latter fallback is the one I was assuming we would be living with for approximately forever - I just forgot for a couple of days about the other fallback code path in pip that doesn't use a PEP 517 backend at all.

[webknjaz]

Is this still relevant? distutils is getting deprecated and is going to be removed from stdlib in CPython 2.12.