New variant in gloo-ee-1.17.0 doesn't consume correct digest

[DreyfussDaena]

Gloo Edge Product

Enterprise

Gloo Edge Version

gloo-ee-1.17.0

Kubernetes Version

v1.28.6

Describe the bug

When setting variant to distroless
we still see the digest of the standard image rather than the distroless image.
This caused the actual pull of the standard image and not the one the distroless ones.

Expected Behavior

When using the distroless we'd expect the image's digest to fit the distroless image.

Steps to reproduce the bug

Update version of gloo-ee, update variant to distroless.

Additional Environment Detail

No response

Additional Context

No response

┆Issue is synchronized with this Asana task by Unito

[DreyfussDaena]

@nfuden Please see that this is dealt with in high priority as this is causing a pushback in our adoption of the new Gloo version.
Thanks a lot!

[nfuden]

Thanks for raising this. We have a lot of pre flight checks but it looks like we need to update our post release runbooks to catch this in addition to more robust release helm checks.

We are looking into it and in the meantime anyone who is using an image other than standard will by default have pods unable to come up due to the hash mismatch.
To work around this in the interim the hash can be scrubbed or looked up from our image repositories.

We are having someone dig into fixing this for 1.17 as well as building stronger checks into our new release pipeline

In the meantime distroless can be enabled by adding the following digests in helm

gloo.gloo.deployment.image.digest=f9b1523055e900436eaa2879f97d159e9b252c268b85a43041809de8410a7ac4
gloo.discovery.deployment.image.digest=20935aae72f6ebb512dc345424dac98ea3c298d88009186632f2d762deded287
gloo.gatewayProxies.gatewayProxy.podTemplate.image.digest=4f6d33ae17eb8899343b57562cbf3f56c28cfdc87f36aae3998fb3cdf7b2ec6b
global.extensions.extAuth.deployment.image.digest=1d30e4928459fb7ffd05942f7b5813bf870fe7ea9201ecf68301f7d3c348e7f6
global.extensions.rateLimit.deployment.image.digest=ee6b0ef6d5e1701539b207c030f1d610f9abe050eca2d0d116c265bf42187262

[davidjumani]

Or just by specifying a blank digest

--set gloo.gloo.deployment.image.digest='' 
--set gloo.discovery.deployment.image.digest='' 
--set gloo.gatewayProxies.gatewayProxy.podTemplate.image.digest='' 
--set global.extensions.extAuth.deployment.image.digest='' 
--set global.extensions.rateLimit.deployment.image.digest=''

[jenshu]

Images affected:

If 1.17.0 was installed with global.image.variant (or any other *.image.variant) set to distroless, fips, or fips-distroless, the following images will have the wrong digest (causing the standard variant digest to be pulled, instead of the desired variant):

distroless images:

quay.io/solo-io/gloo-ee:1.17.0-distroless
quay.io/solo-io/gloo-ee-envoy-wrapper:1.17.0-distroless
quay.io/solo-io/discovery-ee:1.17.0-distroless
quay.io/solo-io/rate-limit-ee:1.17.0-distroless
quay.io/solo-io/extauth-ee:1.17.0-distroless
quay.io/solo-io/observability-ee:1.17.0-distroless       
quay.io/solo-io/kubectl:1.17.0-distroless
quay.io/solo-io/certgen:1.17.0-distroless
quay.io/solo-io/gloo-fed:1.17.0-distroless
quay.io/solo-io/gloo-fed-apiserver:1.17.0-distroless
quay.io/solo-io/gloo-fed-apiserver-envoy:1.17.0-distroless
quay.io/solo-io/gloo-federation-console:1.17.0-distroless

fips images:

quay.io/solo-io/gloo-ee-fips:1.17.0
quay.io/solo-io/gloo-ee-envoy-wrapper-fips:1.17.0
quay.io/solo-io/discovery-ee-fips:1.17.0
quay.io/solo-io/rate-limit-ee-fips:1.17.0
quay.io/solo-io/extauth-ee-fips:1.17.0

fips-distroless images:

quay.io/solo-io/gloo-ee-fips:1.17.0-distroless
quay.io/solo-io/gloo-ee-envoy-wrapper-fips:1.17.0-distroless
quay.io/solo-io/discovery-ee-fips:1.17.0-distroless
quay.io/solo-io/rate-limit-ee-fips:1.17.0-distroless
quay.io/solo-io/extauth-ee-fips:1.17.0-distroless

Workaround

To find the correct image digest for each image, run:

docker manifest inspect <image> -v | jq -r '.Descriptor.digest'

Do a helm upgrade, setting each of the *.image.digest values to the correct digest for the chosen image / variant.

Permanent fix

• We plan to fix the image digests going forward (from release 1.17.1 on).
  • Essentially the fix will be to store the image digest for each variant, in the helm chart. When the chart is rendered, the digest will be chosen based on the selected variant.
• For 1.17.0, the above workarounds will need to be used to get the correct image digests.

[jenshu]

the fixes have been merged and will be available in the next releases, GlooEE v1.18.0-beta1 and v1.17.1

will keep this open until we verify that the correct digests are pulled with the new releases

[sam-heilbron]

@jenshu 1.17.1 has been released. Could you verify that the release fixes the issue as addressed, and share the testing steps you took?

Tangentially, is there a reason we shouldn't just enable digests on PR builds as well, so that when CI runs on PRs it is also testing this? Or digests on nightly builds?

[jenshu]

tested by installing gloo-ee 1.17.1 with each variant (standard, distroless, fips, fips-distroless) and verified pods start successfully, e.g.

helm install -n gloo-system gloo-ee gloo-ee/gloo-ee --create-namespace --set license_key=$GLOO_LICENSE_KEY \
  --version v1.17.1 --set global.image.variant=distroless

also spot checked some of the manifests using the docker manifest inspect command mentioned above, and verified correct image digests were used