Removal of the package registry and storage stages

[jsoriano]

In our view of the Packages Ecosystem, there will be a single public Package Registry in the future. That is, we plan to remove the snapshot and staging registries.
This comes with some challenges, specially to keep backwards compatibility with current versions of Fleet. Let's discuss about its possible problems and their solutions.

As background:

• For the package storage v2 redesign, we are focusing on having a single public registry. This is a common pattern in public registries, as the docker hub or npm.
• In [Change Proposal] Remove release tag from packages package-spec#225, we identified the overlap of the staging and snapshot registries with other ways of declaring the stability of a package.

[jsoriano]

From elastic/package-spec#225 (comment), quoting @joshdover and me:

Please take into account that this situation [making a non-GA package available to GA users] can already happen now without any change. A developer may release a prerelease version according to semver, and forget to change the release tag from ga to beta. This package would appear as stable in current versions of kibana and users could upgrade to it. [...]

Yes, it could happen today but is unlikely due to the fact that we have snapshot and staging registries. Whenever we deprecate/remove those, this situation becomes the norm: all prerelease packages (including dev) will be shown to users on older Kibana versions.

[jlind23]

@jsoriano couldn't we enforce the check of kibana version field here?

[jsoriano]

Regarding the dev packages, I think that here we assume that we need a registry as part of the workflow of a package developer. But we should work in the direction of allowing development only with elastic-package and the elastic stack, without needing a registry. For this, efforts like elastic/elastic-package#550 and elastic/kibana#70582 will help.

[jsoriano]

@jsoriano couldn't we enforce the check of kibana version field here?

Yes, something like this was proposed (see #3666 (comment)), but I would like to avoid the need of this kind of restrictions. We want to encourage developers to support broad ranges of Kibana versions, and we want to simplify things. This kind of restrictions or conventions goes in the opposite direction.

[jsoriano]

Possible enhancement: Put restrictions on versions of the stack that prerelease packages can target, that is that if Kibana implements something like elastic/kibana#122973 in 8.3.0, packages with prerelease tags published in the single registry shouldn't target older versions of Kibana.

Proposed in:

• [Change Proposal] Remove release tag from packages package-spec#225 (comment)
• [Change Proposal] Remove release tag from packages package-spec#225 (comment)

[jsoriano]

Possible enhancement: Add additional backwards compatibility logic in the package registry, when experimental=true is used. This is an option that introduces controlled technical debt because experimental=true is maintained in any case only for backwards compatibility.

Proposed in:

• [Change Proposal] Remove release tag from packages package-spec#225 (comment)
• [Change Proposal] Remove release tag from packages package-spec#225 (comment)

[jsoriano]

Possible enhancement: keep current snapshot and staging registries till the sunset of versions of Kibana that don't have additional filtering capabilities, and don't show prerelease packages in the production registry when experimental=true is used.

Commented in:

• [Change Proposal] Remove release tag from packages package-spec#225 (comment)

[jsoriano]

Possible enhancement: maintain an additional deployment of the package registry v2, to replace current snapshot/staging registries.

This idea was mentioned in the original design document for the package registry v2, and appeared again in elastic/package-spec#225 (comment).

[endorama]

For the package storage v2 redesign, we are focusing on having a single public registry. This is a common pattern in public registries, as the docker hub or npm.

Will it still be possible to use multiple registries? The use case I'm thinking of is using a private registry in addition to the public registry, a common use case (that can help with air-gapped or private deployments)

[jsoriano]

I think there is no support now to use multiple registries. This would be a feature request for Fleet. cc @joshdover.