Add support for bumping docker container versions referenced from within Helm files #5738

brendandburns · 2022-09-16T17:22:38Z

This PR extends the recent Kubernetes support for Docker containers to common Helm chart formats:

foo:
  image:
    repository: sql/sql
    tag: 1.2.3
    registry: docker.io

or alternatively:

foo:
  image:
    repository: sql/sql
    version: 1.2.3

honeyankit · 2022-09-16T18:21:44Z

@brendandburns : The docker CI / CI (docker, docker) (pull_request) build failed. There are few failures due to the naming convention of the method and variable names (Naming/MethodName:). Request you to fix that. Thank you.

brendandburns · 2022-09-16T20:32:58Z

@honeyankit thanks I'll take a look. (is there a way that I can run these locally?)

brendandburns · 2022-09-16T20:41:49Z

@honeyankit style issues addressed (I hope) ptal

jeffwidman · 2022-09-16T21:29:33Z

is there a way that I can run these locally?

https://github.com/dependabot/dependabot-core#running-the-tests describes how to run the Rubocop style checks. I highly suggest running those commands within the docker container as described in the section just above that one.

pavera

This looks good to me. Thanks for all the tests @brendandburns!

bdragon · 2022-09-19T16:41:14Z

docker/lib/dependabot/docker/file_fetcher.rb

-          # in a multi-resource file had better be a valid k8s resource
-          content = ::YAML.safe_load(f.content, aliases: true)
-          likely_kubernetes_resource?(content)
+          if f.type == "file" && f.name == "values.yaml"


Dependabot detects dockerfiles with names like Dockerfile.ci or Dockerfile-test by selecting files that contain /dockerfile/i anywhere in the name.

From my experience it's common for Helm projects to do something similar with multiple values files, like values-staging.yaml, and then helm install -f values-staging.yaml .... Not sure if we'll land on a filename-matching approach that will accommodate the majority of scenarios, but I wanted to bring it up for consideration.

I'm ok to do this. Will add it in a later PR?

Later PR is fine, esp because it will fail "loudly" so we won't forget about it... ie users will complain it's not working 😄

brendandburns · 2022-09-20T19:47:45Z

Rebased.

brendandburns · 2022-09-20T20:00:48Z

@honeyankit this seems to be failing because of a permission problem unrelated to this PR.

If it is something in this PR please let me know and I will try to fix it.

mattt · 2022-09-20T21:48:04Z

@honeyankit this seems to be failing because of a permission problem unrelated to this PR.

If it is something in this PR please let me know and I will try to fix it.

Sorry for the confusion there. That's not a required check, so it's not a blocker for merging.

The workflow in question pushes images built for PRs to GitHub Container Registry, so that we can easily deploy changes before merging. This step currently fails on PRs from forks, because forks won't have write access to GHCR (see #5668). This is a known issue and we're working on a solution.

(Also, thanks for your contribution! I'm really excited to add support in Dependabot for Helm charts)

jeffwidman

👍 in general, had a few trivial questions/suggestions...

docker/lib/dependabot/docker/file_fetcher.rb

docker/lib/dependabot/docker/file_updater.rb

jeffwidman · 2022-09-21T08:01:47Z

docker/spec/fixtures/github/contents_helm_repo.json

+      }
+    }
+  ]
+


Does helm generate these files without a final newline for some reason? I see this is true of several other files as well.

If not auto generated by Helm this way, can you please fix?

Thanks. I filed #5814 to add a linter for this to speed up the feedback loop.

Actually, you mention fixed, but I still see missing newlines on this file and the other json file? Did you forget to git add them perhaps?

So the newlines were there, but I think that because the JSON wasn't fully formatted the newline detector was confused.

I think I have fixed this now.

jeffwidman · 2022-09-21T08:15:09Z

Also, can you please squash the lint change commit into the original commit?

We currently can't merge via squash due to a limitation in our release script so whatever the commit history is on this branch is what makes it into main...

brendandburns · 2022-09-27T19:40:18Z

Comments addressed, commits squashed and rebased.

Please take another look.

Thanks!

docker/lib/dependabot/docker/file_fetcher.rb

brendandburns · 2022-09-28T22:33:55Z

Comment addressed. Please take another look.

bdragon

LGTM. Thanks for the thorough test coverage!

jeffwidman

LGTM other than the missing newline...

Once you fix that, then we are probably ready to 🚢 given all the other ✅ on this PR. 😄

brendandburns · 2022-10-04T22:45:58Z

@jeffwidman

newlines added. fwiw, the JSON formatter for VS Code/Codespaces removes the newline at the end of JSON files, so I'm not sure that newlines for JSON is standard, but nonetheless they should be there now.

Please take another look.

Thanks!

jeffwidman

Thanks!

I have a few other PR's I need to deploy/merge first, but hopefully can push this one in the next day or two.

jeffwidman · 2022-10-05T23:10:48Z

fwiw, the JSON formatter for VS Code/Codespaces removes the newline at the end of JSON files, so I'm not sure that newlines for JSON is standard, but nonetheless they should be there now.

Oh interesting. I'm surprised by that, I'll have to research that a bit more.

jeffwidman · 2022-10-06T02:19:56Z

That looks better, @brendandburns see ☝️ I have no idea what went wrong, something in GitHub's Update with Rebase command... Anyway, I wasn't sure what email you preferred to use, and there was no history due to the force push so I used your @microsoft.com email, as that's what it looks like you used elsewhere on other commits on your profile.

I'll deploy this once CI goes green, and assuming everything looks good in production will merge this later tonight.

brendandburns · 2022-10-06T18:34:44Z

@jeffwidman Thanks and yes, the @microsoft.com address is generally what I use.

Great to see this merge!

nilekhc · 2022-10-11T22:44:16Z

That looks better, @brendandburns see ☝️ I have no idea what went wrong, something in GitHub's Update with Rebase command... Anyway, I wasn't sure what email you preferred to use, and there was no history due to the force push so I used your @microsoft.com email, as that's what it looks like you used elsewhere on other commits on your profile.

I'll deploy this once CI goes green, and assuming everything looks good in production will merge this later tonight.

Hi @jeffwidman, Do we need to cut a new tag to use this feature?

jeffwidman · 2022-10-12T00:12:53Z

It is already deployed to GitHub prod, but requires a feature flag. If you are using it somewhere other than GitHub than you'll need to use mainas not yet on a tagged release.

nilekhc · 2022-10-12T00:15:59Z

It is already deployed to GitHub prod, but requires a feature flag. If you are using it somewhere other than GitHub than you'll need to use mainas not yet on a tagged release.

I am using it outside of GH. When it'll be available on tagged release?

nilekhc · 2022-10-17T17:42:46Z

@jeffwidman Any timeline on when the tag release will cut?

jurre · 2022-10-25T07:27:43Z

I am using it outside of GH. When it'll be available on tagged release?

We don't really do tagged releases anymore as our deployment model has changed a bit and we don't need them anymore, you can just pull the changes in from a sha

nilekhc · 2022-10-25T17:38:06Z

cc @mburumaxwell

jeffwidman · 2022-10-26T09:07:20Z

Long term, I think we probably should still make tagged releases available on RubyGems to make life easier for those who run Dependabot in custom ways. But since we are now deploying multiple times a day straight from PR branches before they merge back to `main` we'll probably move away from manually running the release scripts that push to RubyGems and instead automate it to cut versions on a monthly or weekly basis. This also means we'll likely need to move away from semver guarantees. That said the above is my personal opinion and others on the team may feel differently... So just overall there's some planning / internal discussion that needs to happen and no one has had the bandwidth to drive that yet.. I'm hopeful I might have time before end of quarter but I can't guarantee it.

jeffwidman · 2022-10-28T07:11:27Z

Currently this is buried behind a feature flag for folks on GitHub.com...

We'd like to start slowly rolling it out for everyone, but before I do that, I'd like to test on a few repos to ensure no unexpected issues. I can / will setup a fake test repo, but there's nothing like live data to confirm things.

Anyone watching this PR care to volunteer to have their repo(s) be part of the beta test?

Repos must match the following criteria:

on GitHub.com
public, because for security reasons we can't access private repos w/o jumping through some hoops
have helm files... ideally with a combination of up-to-date and outdated docker image tags

If interested, please comment on this thread with the URLs of any repos you own/maintain.

jeffwidman · 2022-10-31T16:48:19Z

@nilekhc This is now live on RubyGems: https://rubygems.org/gems/dependabot-omnibus/versions/0.213.0

nilekhc · 2022-10-31T17:40:53Z

Currently this is buried behind a feature flag for folks on GitHub.com...

We'd like to start slowly rolling it out for everyone, but before I do that, I'd like to test on a few repos to ensure no unexpected issues. I can / will setup a fake test repo, but there's nothing like live data to confirm things.

Anyone watching this PR care to volunteer to have their repo(s) be part of the beta test?

Repos must match the following criteria:

on GitHub.com

public, because for security reasons we can't access private repos w/o jumping through some hoops

have helm files... ideally with a combination of up-to-date and outdated docker image tags

If interested, please comment on this thread with the URLs of any repos you own/maintain.

Hey @jeffwidman, we can test with our repo if you are still looking for volunteers.
https://github.com/Azure/secrets-store-csi-driver-provider-azure

Let me know what I need to do.

cc @aramase

jeffwidman · 2022-11-03T03:17:12Z

Awesome thanks @nilekhc

I will enable the feature flag on your repo tomorrow, and then since it's a public repo I think I can internally trigger a dry-run to test it. I'll post a follow-up once the flag is enabled.

jeffwidman · 2022-11-03T03:19:53Z

Also @nilekhc you'll need to enable a "docker" ecosystem check in https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/.github/dependabot.yml for the whatever folder holds these files. IIRC, the check will recursively search for files, so if you've got nested subdirs of charts, targeting the "docker" check at the parent dir should be fine.

nilekhc · 2022-11-04T05:36:57Z

Also @nilekhc you'll need to enable a "docker" ecosystem check in https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/.github/dependabot.yml for the whatever folder holds these files. IIRC, the check will recursively search for files, so if you've got nested subdirs of charts, targeting the "docker" check at the parent dir should be fine.

PR - Azure/secrets-store-csi-driver-provider-azure#1014

nilekhc · 2022-11-11T04:10:31Z

@jeffwidman It's working on our repo -
https://github.com/Azure/secrets-store-csi-driver-provider-azure/pulls

jeffwidman · 2022-11-11T05:15:10Z

Awesome news thanks for circling back!

jeffwidman · 2022-11-15T07:23:07Z

I've enabled the feature flag on 100% of GitHub.com cloud repos, so anyone is now welcome to try this on their repos. If you run into any bugs, please file an issue.

nilekhc · 2022-12-01T01:12:22Z

I've enabled the feature flag on 100% of GitHub.com cloud repos, so anyone is now welcome to try this on their repos. If you run into any bugs, please file an issue.

@jeffwidman Did you do any special setup while enabling this feature? I am trying to run it manually by building dependabot docker image and running dependabot/dependabot-core bundle exec ruby ./generic-update-script.rb but got error Found neither Kubernetes YAML nor Dockerfiles in <directory_name> (Dependabot::DependabotError). <directory_name> here is pointing to the directory where I have values.yaml file.

jeffwidman · 2023-01-11T06:31:37Z

☝️ is resolved, right @nilekhc ? Catching up on old notifications... 😀

nilekhc · 2023-01-12T00:32:17Z

☝️ is resolved, right @nilekhc ? Catching up on old notifications... 😀

It is!

brendandburns requested a review from a team as a code owner September 16, 2022 17:22

brendandburns force-pushed the helm branch from 30cb2b0 to 06d24d0 Compare September 16, 2022 17:23

pavera approved these changes Sep 19, 2022

View reviewed changes

bdragon reviewed Sep 19, 2022

View reviewed changes

honeyankit self-requested a review September 19, 2022 18:16

honeyankit approved these changes Sep 19, 2022

View reviewed changes

jeffwidman reviewed Sep 21, 2022

View reviewed changes

brendandburns force-pushed the helm branch from 919d15d to a4d99e6 Compare September 27, 2022 19:37

bdragon reviewed Sep 28, 2022

View reviewed changes

docker/lib/dependabot/docker/file_fetcher.rb Outdated Show resolved Hide resolved

brendandburns force-pushed the helm branch 3 times, most recently from 70e9e0c to a5152a6 Compare September 28, 2022 22:33

bdragon approved these changes Sep 29, 2022

View reviewed changes

jeffwidman mentioned this pull request Sep 29, 2022

A linter should flag missing newlines at end of files #5814

Open

1 task

jeffwidman reviewed Sep 29, 2022

View reviewed changes

brendandburns force-pushed the helm branch 2 times, most recently from 65a8ef3 to 6f6a533 Compare October 4, 2022 22:44

jeffwidman approved these changes Oct 5, 2022

View reviewed changes

Add support for helm files.

7046a3c

jeffwidman force-pushed the helm branch from 41e62ac to 7046a3c Compare October 6, 2022 02:16

jeffwidman merged commit 96cbb6c into dependabot:main Oct 6, 2022

nilekhc mentioned this pull request Oct 11, 2022

Support for helm tinglesoftware/dependabot-azure-devops#366

Closed

jeffwidman mentioned this pull request Oct 31, 2022

Update Docker image on kubernetes deployment files #2217

Closed

pavera mentioned this pull request Oct 31, 2022

v0.213.0 #6002

Merged

jeffwidman mentioned this pull request Nov 29, 2022

Support for HelmRelease #1744

Open

jeffwidman changed the title ~~Add support for helm files.~~ Add support for bumping docker container versions referenced from within Helm files Nov 29, 2022

slonka mentioned this pull request Sep 11, 2023

Add helm directory to dependabot updates kumahq/kuma#7657

Open

Add support for bumping docker container versions referenced from within Helm files #5738

Add support for bumping docker container versions referenced from within Helm files #5738

Conversation

brendandburns commented Sep 16, 2022

honeyankit commented Sep 16, 2022

brendandburns commented Sep 16, 2022

brendandburns commented Sep 16, 2022

jeffwidman commented Sep 16, 2022

pavera left a comment

Choose a reason for hiding this comment

bdragon Sep 19, 2022

Choose a reason for hiding this comment

brendandburns Sep 20, 2022

Choose a reason for hiding this comment

jeffwidman Sep 21, 2022 • edited Loading

Choose a reason for hiding this comment

brendandburns commented Sep 20, 2022

brendandburns commented Sep 20, 2022

mattt commented Sep 20, 2022 • edited Loading

jeffwidman left a comment

Choose a reason for hiding this comment

jeffwidman Sep 21, 2022

Choose a reason for hiding this comment

brendandburns Sep 27, 2022

Choose a reason for hiding this comment

jeffwidman Sep 29, 2022

Choose a reason for hiding this comment

jeffwidman Sep 29, 2022

Choose a reason for hiding this comment

brendandburns Oct 4, 2022

Choose a reason for hiding this comment

jeffwidman commented Sep 21, 2022

brendandburns commented Sep 27, 2022

brendandburns commented Sep 28, 2022

bdragon left a comment

Choose a reason for hiding this comment

jeffwidman left a comment

Choose a reason for hiding this comment

brendandburns commented Oct 4, 2022

jeffwidman left a comment

Choose a reason for hiding this comment

jeffwidman commented Oct 5, 2022

jeffwidman commented Oct 6, 2022

brendandburns commented Oct 6, 2022

nilekhc commented Oct 11, 2022

jeffwidman commented Oct 12, 2022

nilekhc commented Oct 12, 2022

nilekhc commented Oct 17, 2022

jurre commented Oct 25, 2022

nilekhc commented Oct 25, 2022

jeffwidman commented Oct 26, 2022 via email • edited Loading

jeffwidman commented Oct 28, 2022

jeffwidman commented Oct 31, 2022

nilekhc commented Oct 31, 2022

jeffwidman commented Nov 3, 2022

jeffwidman commented Nov 3, 2022

nilekhc commented Nov 4, 2022

nilekhc commented Nov 11, 2022

jeffwidman commented Nov 11, 2022

jeffwidman commented Nov 15, 2022

nilekhc commented Dec 1, 2022

jeffwidman commented Jan 11, 2023

nilekhc commented Jan 12, 2023

jeffwidman Sep 21, 2022 •

edited

Loading

mattt commented Sep 20, 2022 •

edited

Loading

jeffwidman commented Oct 26, 2022 via email •

edited

Loading