Add 'setuptools.sub_command' entry-points

[abravalheri]

This PR expands on the work done in #2591, to support plugins that wish to add custom steps to the build process.

Summary of changes

The method setuptools.Command.get_sub_commands is extended to load sub-commands from setuptools.sub_commands.* entry-points. (It also automatically insert items to distribution.cmdclass, so there is no need to specify extra distutils.commands entry points).

The entry-points should correspond to a setuptools.Command subclass when loaded.

There are 3 optional attributes these subclasses can define to customise when/if the sub-commands will run:

• after OR before -- a string with the name of another already defined sub-command (by default the new sub-command will run after the last of the existing ones).
• condition -- a string with the name of a predicate method in the parent command class.

Closes (potentially?) #2591

Things purposefully not considered (future work)

The changes in this PR improve the ergonomics for setuptools plugins writers, but not for end-users (via setup.py or setup.cfg files).

Currently there is an asymmetry on how plugin writers and end-users can customize the setuptools build process. Plugin writers can define any entry-point, while end users are restricted to cmdclass.
I believe that this asymmetry should be considered in a separated proposal, and then, the ergonomics defined here can be extended to end-users.

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d/.
  (See documentation for details)

[jaraco]

disclaimer: the following feedback is made without any in-depth analysis of the problem or solution.

• after OR before -- a string with the name of another already defined sub-command (by default the new sub-command will run after the last of the existing ones).

When I see after and before, it feels like an imperfect ordering with possible irreconcilable parameters. How about instead allowing for a priority or order that's a numeric value, allowing plugins to declare a position in an ordered set?

sub_commands

As a nitpick, let's use subcommand or some other single-word moniker if possible (at least in the public interface). Having an underscore-separated indicator (sub_commands) creates ambiguity (get_sub commands vs. get sub_commands). Maybe step or operation.

[abravalheri]

Hi @jaraco thank you very much for the comments and sorry for the long time it took me to reply.

When I see after and before, it feels like an imperfect ordering with possible irreconcilable parameters. How about instead allowing for a priority or order that's a numeric value, allowing plugins to declare a position in an ordered set?

Do you mean having 2 parameters? One for the priority order (i.e. the order with which the plugins will be loaded) + one for the position? I think that even if we add such kind of parameter there will always be the chance that the position in the ordered set is not consistent across multiple environments.

Depending on which dependencies the final user adds to their [build-system] requires (e.g. if they add multiple plugins defining extra build-steps), plugins will experience different orderings.
In my opinion the best we can do is to guarantee the relative position of the 3rd party in relation to the "built-in build steps".

Would it be better if instead before or after we have append or prepend, meaning that the command should be run after all "builtin build steps" or before all of them?

As a nitpick, let's use subcommand or some other single-word moniker if possible (at least in the public interface). Having an underscore-separated indicator (sub_commands) creates ambiguity (get_sub commands vs. get sub_commands). Maybe step or operation.

I agree with you in this point, however distutils already uses sub_commands and get_sub_commands, so the implementation in the PR tried to promote consistency. I am happy to change that if you prefer.

[abravalheri]

Alternatively we can give up on adding another type of entry-point and recommend for users to implement the plugins using a combination of distutils.commands (to define the custom build steps) and setuptools.finalize_distribution_options (to insert the new commands into the subcommand list in arbitrary positions).

This approach seems to be viable nowadays (see https://github.com/abravalheri/experiment-setuptools-plugin).

[plannigan]

I feel like it isn't possible to guarantee any consistent complete ordering when considering multiple plugins that don't know about each other. finalize_distribution_options support an order propery to specify a preferred priority. But there is no mention of how setuptools breaks ties. This would leave us in the same place as being discussed here, which seems OK. It seems unlikely that there would be two plugins would have an interaction where one doesn't operate correctly when running after the other. However, I could see a plugin that specifically needs to be run before build_clib, as an example. So I guess I'm leaning towards supporting both before/after and priority.

before & after can be used for when the plugin specifically needs to be executed relative to first party setuptools sub-commands. This allows a plugin to control what bucket they are placed in. A plugin that registers for after X is in a distinct bucket that always executes before a plugin that registers for before x+1. If you "just want to run when build executes", don't specify anything and it will be placed in the last bucket (maybe a special bucket that comes after the after bucket of the last first party sub-command?).

order (used to be consistent with finalize_distribution_options) defaults to 0 and is only used to sort the ordering of custom sub-commands within each specific bucket.

[jaraco]

In the keyring project, the plugins specify a priority (with some coded heuristics about the meaning) and some of them use the priority of another to specify a relative priority. That approach has its downsides, but it does provide a straightforward way to order the plugins.

I don't feel strongly about it. Happy to proceed with whatever approach Anderson prefers.

[abravalheri]

Hi @jaraco, after giving some thought to this problem, I decided to modify the PR in the following ways:

• Renamed the entry-point from setuptools.sub_commands.build to setuptools.build_steps after your comment.

• Added a priority customization. This attribute is used to sort the entry-points before they are added to the list of build-subcommands. Defaults to 0.
  If entry-points have the same name, the one with higher priority wins.

• Removed the confusing after and before attributes.

• Added an insert_build_step customization.
  By default, custom build steps are appended to the build.sub_commands list.
  Higher priority commands will be appended "first".
  This should cover the majority of use cases.

  If this is not enough for a specific use case, developers can implement the insert_build_step class/static method to fine tune in which order the custom build step will be executed.

Please feel free to bike-shed this names, I am not very sure myself.