Move Black into a plugin to enable turning it on and off via backend_plugins

[Eric-Arellano]

Problem

We need some mechanism to turn off autoformatters and linters in V2 other than the current --fmt-{isort,scalafmt,etc}-skip, which doesn't work because that uses a task-specific option system.

Ideally, the option will be tool-specific, so you can say something like:

[black]
enable: True

[mypy]
enable: False

However, the solution should also be hygienic. If, for example, someone doesn't want to use Black, then we don't want Black's rules to pollute the rule graph. Nor do we want to make rule authors provide a no-op implementation.

Solution

We decided in #8346 to turn each linter into its own plugin, meaning it may be enabled via the backend_packages option like this:

[GLOBAL]
backend_packages: +[
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.isort",
    "pants.backend.scala.lint.scalafmt",
  ]

When specified, that plugin will be enabled, meaning that its options may be configured and its rules will be automatically registered against fmt and lint.

Result

We have a scalable and explicit mechanism for users to enable formatters and linters in a way that also makes it effortless for rule authors to have a way to opt-out of their tool.

This only implements Black. In a followup, we will implement isort under the plugin pants.backend.python.lint.isort.

For now, V1 tasks will keep the same mechanism as before. We could move them towards this new plugin system, but save that for a followup if we choose to make the deprecations necessary to move V1 users towards the new system.

[jsirois]

Just a note, but backend_packages as things currently stand is a black hole / requires a Pants PHD. Unlike published plugins, which are discoverable on PyPI, these things can only be learned about by browsing the codebase. Additionally, each plugin implemented as a backend_package shipped with pants (especially when disabled by default), is a semver footgun. Its very easy to refactor these packages names and be un-aware that this is a user-interface break.

[cosmicexplorer]

Good change!

[stuhood]

Agreed with John: re "backend packages" vs "plugins". If we could lower the overhead of plugins and just do things that way, that would be ideal. But this nonetheless feels like a step forward (especially if we think that we can preserve the package name across a transition from "backend package" to "plugin", which it sounds like we could if we dropped the contrib nomenclature).

[cosmicexplorer]

Another set of thoughts at #8686!

[Eric-Arellano]

Agreed with John: re "backend packages" vs "plugins". If we could lower the overhead of plugins and just do things that way, that would be ideal. But this nonetheless feels like a step forward

Agreed with this all. I think there is still much room to improve this user experience, especially with discoverability. But, this is a good foundation (much better than the --black-enable proposal).

An additional open question is how to approach migrating towards this approach for V1? Do we want to deprecate the current mixed approach of sometimes being auto-installed and having to disable via --{fmt,lint}-skip + sometimes having to download a contrib package and configure it via backend_packages? The deprecation would look like this:

1. If not using skip: True, i.e. using the default or skip: False, deprecate that they should instead set pants.backend.python.lint.isort etc. This means people who want the plugin will now be using it the correct way. At the end of this same deprecation cycle, change the default of skip to be True so that people won’t need to have anything in pants.ini to ignore the plugin
2. Then, deprecate --skip entirely, as it’s now unnecessary. We skip by default. If you do not want to skip, then you should do via registering the plugin.

[benjyw]

Just a note, but backend_packages as things currently stand is a black hole / requires a Pants PHD. Unlike published plugins, which are discoverable on PyPI, these things can only be learned about by browsing the codebase. Additionally, each plugin implemented as a backend_package shipped with pants (especially when disabled by default), is a semver footgun. Its very easy to refactor these packages names and be un-aware that this is a user-interface break.

I think backend_packages should come into the light, and codebases should have to use it to explicitly opt in to every piece of pants functionality (including Python, JVM etc.) That is, there should be no autodiscovery of a top-level register.py in a plugin.

[This is separate from, but related to, the question of whether any backends ship in the core pantsbuild.pants artifact or not (I lean towards "not").]

Basically, I think will make sense to publish multiple backends (i.e., register.pys), in a single plugin (i.e., pypi project). So for example we can publish multiple linters in pantsbuild.pants.python.lint on pypi, but users will have to opt in to the linters they want using backend_packages. Similarly for codegen, for scala vs java (although in that case it might make more sense to actually have two plugins), and so on. There will obviously be some plugins that only contain a single backend, but I don't see a good reason to allow for autodiscovery in that special case. What we gain in convenience we more than lose in obfuscation.