abstracting away plugin loading from the pythonpath a little

[cosmicexplorer]

In response to this comment from @jsirois at #8683 (comment):

Unlike published plugins, which are discoverable on PyPI, these things can only be learned about by browsing the codebase.

This has consistently represented an annoying amount of confusion for me. My first thought was to consider moving them all to published plugins, which isn't necessarily a bad idea since by definition these are optional code. However, thinking about the idea introduced in #8683:

[black]
enable: True

[mypy]
enable: False

It would be really nice to be able to have these as options on a per-plugin basis, but that doesn't immediately work with the way plugins are loaded, since the [black] scope presumably is only available when the plugin is loaded. That being said, we could consider (perhaps automatically) introducing an option that (for now) determines whether the rest of the plugin loads, so like in a register.py:

plugin_name = 'black'

pants.ini:

[black-plugin]
enable: False

I looked in extension_loader.py for a bit, and got as far as https://github.com/pantsbuild/pants/compare/master...cosmicexplorer:extension-loader-optional-plugin-name?expand=1 before realizing we would need to create a whole new BuildConfiguration to be able to do that, and decided that it might be better as a global list option, something like (in pants.ini):

plugins_from_pythonpath: [..., 'black', 'mypy', 'some-loose-source-plugin-in-my-own-repo']
backend_packages: [...]

Which we would check plugins' plugin_name against. But I'm not sure that's the right name, or where exactly to go with this option.

[cosmicexplorer]

Another possibility -- scan pants.ini for scopes named [*-plugin], and then somehow delete those scopes before options parsing, instead of having to manually add entries to plugins_from_pythonpath. But then that wouldn't be configurable via command-line like other options, which I don't like.

[thejcannon]

I dont think this issue is really relevant with Pants today, as the docsite helps people know what's built into Pants and therefore makes "backend" packages discoverable.