Moving OCI integration from experimental to full feature

[scottrigby]

Most of the work outlined in hip-0006 has been finished.

The upstream ORAS Go project that Helm imports now has a stable 1.0.0 release, with a backward compatibility guarantee. This was the last major step in unblocking releasing Helm OCI integration as a full feature.

Remaining steps

• Documentation only: Helm v3.7.0 OCI Helm push to ecr gives 404 Not found #10169 (comment)
  • fix(proto): remove unused fields #1249
• OCI registry being treated as Helm registry with failed update #10534
  • Fixing issue where OCI handling early causes a bad message #10535
• Adopt Helm best practice for altering build affix in versions when pushing to OCI #10166
  • For OCI, convert + to _ before pull, and + to _ before push #10521
• helm registry login cannot use existing config file for authentication #10156
  • Enable fallback on Docker config using custom auth file oras-project/oras-go#92
  • registry: change default registry config from registry.json to registry/config.json #10536
• OCI: no version range support for dependencies/helm install #9694
  • Backport of tag listing to v1 oras-project/oras-go#89
  • OCI version range support #10527
• Move internal/experimental/registry out of internal package #9188
  • Move oci out experimental #10546 to be merged last, after all other work above is complete

Not identified as blockers

However, if someone has time to test these oci-related PRs, they may still get into 3.8.0.

• How can I push chart to an insecure OCI registry with helm v3 #6324
  • PR ready for review 👀: Push to OCI insecure registry #10408
• Support for "Uploader Plugins" #10175
  • PR ready for review 👀 Add support for uploader plugins #10430

Follow-up

This issue is only concerned with comprehensively listing any blockers for moving OCI support to a full feature. New features can be added after it is moved from experimental. After that point, only backward-compatible changes will be accepted until Helm 4.

• Join the discussion about backward-compatible features here: What do you want to see in OCI Support? #10312

Update (by mattfarina) 12/3: Connecting with technosophos and jdolitsky we reviewed the open issues from bacongobbler and added two additional tasks to the remaining steps.

[bacongobbler]

Anything else?

Here's my laundry list... I started by searching the issue queue. Here's the search terms I used to look for open tickets:

https://github.com/helm/helm/issues?q=is%3Aissue+is%3Aopen+oci
https://github.com/helm/helm/issues?page=1&q=is%3Aissue+is%3Aopen+docker
https://github.com/helm/helm/issues?q=is%3Aissue+is%3Aopen+registry

This was just a casual search... It might be worth triaging the entire queue just to make sure no stone is left unturned.

Anyways, I saw a few tickets that might be worth evaluating (some have been discussed at length during previous developer calls):

• Add alias support for OCI Charts #10013
• OCI: no version range support for dependencies/helm install #9694
• How can I push chart to an insecure OCI registry with helm v3 #6324
• Helm 3: 'helm chart push/pull' can't fetch oauth token #6214
• helm registry login cannot use existing config file for authentication #10156
• Helm login timeout issue - deploying over high latency connections #10123
• Adopt Helm best practice for altering build affix in versions when pushing to OCI #10166
• helm pull/install/upgrade with oci registry randomly fail with 401 Unauthorized #10396

#9694 in particular seems like a critical code path that should be addressed. No version range support for both helm install and dependencies is a big regression compared to chart repositories because users must now know what version they need to install. helm install wordpress --generate-name is currently not possible with OCI. Users MUST declare what version to install. To me that's a huge regression.

I know we've discussed helm search at great length during a few developer calls, but we need an answer whether the maintainers are going to work on a solution, or state that helm search does not support OCI registiries (and mark that as a ❌when comparing to chart repositories). Choosing to implement it could affect how information is stored in the registry, which would constitute a breaking change.

#10166 might also be worth taking a closer look as it pertains to how chart versions are stored in the registry.

I'd also recommend a bake-off between what you can do with chart repositories vs OCI registries before we can mark it as graduated. If only to inform users of the benefits and tradeoffs they make when choosing one or the other.

https://github.com/helm/helm/blob/main/CONTRIBUTING.md#semantic-versioning states that "Chart repository functionality MUST be backward compatible". But it doesn't state anything about OCI registry functionality.

Finally... The Helm documentation on chart registries would also be worth taking a closer look to make sure everything is up to date. Things like "why someone would choose OCI registries over chart repositories" would be worth documenting.

[bacongobbler]

Doc suggestion: "migrating charts from a chart repository to OCI". Either as a plugin similar to https://github.com/helm/helm-2to3 or just a guide on what commands to run.

[bacongobbler]

I'd also recommend looking at the oras issue queue, searching for any open tickets that may be worth fixing upstream.

This one seems related to #9694: oras-project/oras-go#19

[pytimer]

Implement helm push/pull support insecure OCI registry in #10408, fix How can I push chart to an insecure OCI registry with helm v3 #6324

[scottrigby]

Updates:

Folks are working hard on these issues leading up to and potentially during the holidays. In order for this to be moved from experimental to full feature in the next minor release (3.8.0) all blockers would have to be complete by the scheduled RC date (Jan 10, 2022). If not, we can re-target to the next minor release (3.9.0). Let's see what we can accomplish!

I have removed "Forthcoming issue: @jdolitsky re DOCKER_CONFIG" from our checklist in the issue description, and added a comment here with the current plan: #10156 (comment) @jdolitsky you can cross this off your list to explain in an issue ✨

@pytimer I have added #6324 and your PR #10408 to a new category above called "Not yet identified as a blocker", since I'm not yet clear whether this should be a blocker for moving OCI from experimental to full feature, or a nice to have by then and alternatively could be a follow-up. Will see if we can get this reviewed.

[pytimer]

@scottrigby Thanks, i know. :-) If something should i do, please tell me.

[KlavsKlavsen]

I assume target of 3.8.0 was not reached with this.. So 🤞 for 3.9.0 :)

[scottrigby]

@KlavsKlavsen today's RC was pushed back two days to accommodate completing this. This still gives a full week between RC and 3.8.0 to fix regressions. It is still possible we won't make the deadline, but a number of us are putting in a lot of work to make this happen, and it's looking very good right now.

[scottrigby]

Update: we are nearly there!

• OCI version range support #10527 PR is solid. New test added. All checks pass. Just awaiting one other Helm maintainer to review/approve then I will merge
• Oras maintainers can then cut oras-go v1.1.0, since we now confirmed tag API, auth, and docker config fallback all work now in v1.1.0-rc3
• Last step will be to finish moving the package out of experimental, remove the experimental env var, and feature flag gates. Thanks to @runyontr for starting this – @dperaza4dustbit is planning to bring in his commits into a separate PR and finish that tomorrow. If for some reason David is pulled away, I can do that.

How you can help: test the RC!

Assuming we will get this into the Helm RC, we will want everyone interested in this feature to test it. There will be a week to fix regressions found before 3.8.0. Will confirm in this issue if/when that is indeed the case.

We are looking good! 🤞 will keep issue updated

[danquah]

Awesome! Just say the word, I'll be happy to throw some hours into testing out a project we have that currently uses the experimental flag to push some Helm Charts to both Github Packages and Azure Container Registry

[KlavsKlavsen]

link to Helm RC - and we'll update our docker images and CI pipelines to use this for OCI and see how it works

[scottrigby]

This is complete! Closing this meta tracking issue, and will add a comment once RC is out. Thank you all for your efforts, and thanks in advance for testing the oci feature in the RC ✨

[scottrigby]

📣 v3.8.0-rc.1 is out! Please test and promote widely for the most oci-related user regression testing possible before the 3.8.0 milestone on 1/19 .