Multiple helm chart updates

[kaosmonk]

Hi guys,

My keel setup is acting a bit and I can't find out why.

I have deployed my service through a helm chart. This is my values.yaml for that service:

keel:
  policy: force
  match-tag: true
  pollSchedule: "@every 3m"
  images:
    - repository: image.repository
      tag: image.tag

and since I'm at an early stage and do not yet have semver tags, my docker image gets master tag after each build. Which is why I had to set

      annotations:
        date/deploy-date: {{ now | quote }}

in my service deployment.yaml file.

And that works, keel is subscribed to pubsub events and I can see the messages coming in whenever a new image with the master tag is being built.

But something must not be set right since keel runs two deploys for each build. Here's what it says in the log:

time="2018-08-31T12:46:23Z" level=info msg="provider.kubernetes: processing event" registry= repository=gcr.io/PROJECT/REPO tag=abc98560f2716e14ca03d26979e25fe7018dc3bd
time="2018-08-31T12:46:23Z" level=info msg="provider.helm: processing event" registry= repository=gcr.io/PROJECT/REPO tag=abc98560f2716e14ca03d26979e25fe7018dc3bd
time="2018-08-31T12:46:23Z" level=info msg="provider.kubernetes: no plans for deployment updates found for this event" image=gcr.io/PROJECT/REPO tag=abc98560f2716e14ca03d26979e25fe7018dc3bd
time="2018-08-31T12:46:23Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=abc98560f2716e14ca03d26979e25fe7018dc3bd
time="2018-08-31T12:46:23Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=abc98560f2716e14ca03d26979e25fe7018dc3bd
time="2018-08-31T12:46:23Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=abc98560f2716e14ca03d26979e25fe7018dc3bd
time="2018-08-31T12:46:24Z" level=info msg="provider.kubernetes: processing event" registry= repository=gcr.io/PROJECT/REPO tag=master
time="2018-08-31T12:46:24Z" level=info msg="provider.kubernetes: no plans for deployment updates found for this event" image=gcr.io/PROJECT/REPO tag=master
time="2018-08-31T12:47:00Z" level=info msg="provider.helm: release updated" release=RELEASE version=9
time="2018-08-31T12:47:00Z" level=info msg="provider.helm: processing event" registry= repository=gcr.io/PROJECT/REPO tag=master
time="2018-08-31T12:47:00Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=master
time="2018-08-31T12:47:00Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=master
time="2018-08-31T12:47:00Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=master
time="2018-08-31T12:47:28Z" level=info msg="provider.helm: release updated" release=RELEASE version=10

So I'm building one image that I tag with both BRANCH_NAME (master in this case) and COMMIT_HASH (abc98560f2716e14ca03d26979e25fe7018dc3bd) and it seems like keel deploys both of them? Why would that be the case?

Any ideas?

Thanks!

[rusenask]

Hi, looking at Keel config, match-tag is not available in the keel config:

type KeelChartConfig struct {
	Policy               types.PolicyType  `json:"policy"`
	Trigger              types.TriggerType `json:"trigger"`
	PollSchedule         string            `json:"pollSchedule"`
	Approvals            int               `json:"approvals"`        // Minimum required approvals
	ApprovalDeadline     int               `json:"approvalDeadline"` // Deadline in hours
	Images               []ImageDetails    `json:"images"`
	NotificationChannels []string          `json:"notificationChannels"` // optional notification channels
}

therefore it probably runs an update for your COMMIT_HASH image too

[kaosmonk]

Hi,

So in order to get only master tag deployed, I need to take it out? I'll try it right away!

Thanks for the fast response! Will let you know if it works shortly!

[rusenask]

yeah, or use plain k8s manifests instead of helm :)

[kaosmonk]

Hm no, that didn't help. It still runs the update twice in a row.

Any other ideas? I run out of mine.

[kaosmonk]

Is there a way to match a tag by specifying a regex?

[kaosmonk]

If I swap policy: force with policy: all helm chart/deployment is not being updated at all. Which is expected behaviour.

[rusenask]

Can you upload the logs when there's only one image pushed but you get two updates? not sure how that could happen.

All policy is the same as major, only applicable for semver. Regex matching is something that we want to add :)

[kaosmonk]

Sorry but what logs? Keel logs? There's nothing but what I already posted above. The lines are always the same.

I'm building a single docker image in GCB that gets tagged with BRANCH_NAME and GIT_HASH and then pushed to GCR. Keel picks up the events in pubsub and run the deploys/helm upgrades. And I see the deployment revisions and correct docker images being deployed.

I'm also confused to why it deploys twice...

[rusenask]

what do you mean by "tagged with BRANCH_NAME and GIT_HASH", wouldn't that be two images?

[kaosmonk]

Well I'm building a single image from the master branch that gets multiple tags applied and as such pushed to GCR.

[rusenask]

Okay, so that's where two deploys come since both tags get individual events.

Just made a small fix by adding matchTag to the config and checking it.

keel:
  policy: force
  matchTag: true   # <--- 
  pollSchedule: "@every 3m"
  images:
    - repository: image.repository
      tag: image.tag

Could you try out keel's alpha tag image? If it's working as expected, I will make a PR.

docker push keelhq/keel:alpha
The push refers to repository [docker.io/keelhq/keel]
3ba8a7687f96: Pushed 
c86d292c7c48: Layer already exists 
cd7100a72410: Layer already exists 
alpha: digest: sha256:89a7b50add0c21d5480546f17c7d5701e92abf96baf234ed50679697baa04886 size: 949

[kaosmonk]

Will try it out in a moment! Thanks!

I wonder why am I getting 2 deployments when only one tag for that image matches master (which my service got deployed from and is set in deployment.yaml) and the other one is a git hash...

[rusenask]

with force policy any pushed tag will update, so you could be pushing a, b, c tags and every time it would update it. Or if you just pushed GIT_HASH tag it would also update the deployment, that's why we added matchTag to the kubernetes provider (but it wasn't ported to the helm provider).

[kaosmonk]

alpha is giving me this

ERROR: logging before flag.Parse: E0901 10:05:03.845283       1 reflector.go:205] github.com/keel-hq/keel/internal/k8s/watcher.go:49: Failed to list *v1beta1.CronJob: cronjobs.batch is forbidden: User "system:serviceaccount:keel:keel" cannot list cronjobs.batch at the cluster scope: Unknown user "system:serviceaccount:keel:keel"
time="2018-09-01T10:05:04Z" level=info msg="trigger.pubsub: subscribing for events..." subscription=keel-proj-poc-cluster-proj-111222-gcr topic=gcr```

[rusenask]

You need to update chart (https://github.com/keel-hq/keel/pull/262/files). In a previous version cron job support was added.

[kaosmonk]

Yup, just noticed that myself. On my way!

[kaosmonk]

It works as advertised now!

time="2018-09-01T10:38:06Z" level=info msg="provider.helm: processing event" registry= repository=gcr.io/PROJECT/REPO tag=222490c3640b9e7e6213a474edc8bd9961ad5452
time="2018-09-01T10:38:06Z" level=info msg="provider.kubernetes: processing event" registry= repository=gcr.io/PROJECT/REPO tag=222490c3640b9e7e6213a474edc8bd9961ad5452
time="2018-09-01T10:38:06Z" level=info msg="provider.kubernetes: no plans for deployment updates found for this event" image=gcr.io/PROJECT/REPO tag=222490c3640b9e7e6213a474edc8bd9961ad5452
time="2018-09-01T10:38:06Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=222490c3640b9e7e6213a474edc8bd9961ad5452
time="2018-09-01T10:38:06Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=222490c3640b9e7e6213a474edc8bd9961ad5452
time="2018-09-01T10:38:06Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=222490c3640b9e7e6213a474edc8bd9961ad5452
time="2018-09-01T10:38:06Z" level=info msg="provider.helm: match tag set but tags do not match, ignoring" parsed_image_name="gcr.io/PROJECT/REPO:master" policy=force target_image_name=gcr.io/PROJECT/REPO
time="2018-09-01T10:38:06Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=222490c3640b9e7e6213a474edc8bd9961ad5452
time="2018-09-01T10:38:07Z" level=info msg="provider.helm: processing event" registry= repository=gcr.io/PROJECT/REPO tag=master
time="2018-09-01T10:38:07Z" level=info msg="provider.kubernetes: processing event" registry= repository=gcr.io/PROJECT/REPO tag=master
time="2018-09-01T10:38:07Z" level=info msg="provider.kubernetes: no plans for deployment updates found for this event" image=gcr.io/PROJECT/REPO tag=master
time="2018-09-01T10:38:07Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=master
time="2018-09-01T10:38:07Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=master
time="2018-09-01T10:38:07Z" level=error msg="provider.helm: failed to parse version" error="invalid semantic version" tag=master
time="2018-09-01T10:38:42Z" level=info msg="provider.helm: release updated" release=APP version=2

Single update and the correct one, with the matching tag! Thank you for fast response and the fix!