kubernetes: query docker registry for images before creating Pod

[cognifloyd]

Description

Pull policies match the synchronous nature of the Docker runtime's API calls. In pipeline yaml, this is defined under pull: for steps, services, and secrets.

The Kubernetes runtime maps this to the Kubernetes imagePullPolicy. These get mapped like this:

Vela pull:	k8s ImagePullPolicy	Match?
always	Always	❗ not really a match! (even though they look like it)
never	Never	✔️
not_present	IfNotPresent	✔️
on_start	Always	❓ apparent mismatch (but actually the same)

Note the apparent mismatch between on_start and Always. This is the code comment explaining why this is acceptable:

   // if the pipeline container image should be pulled on start, than
   // we force Kubernetes to pull the image on start with the always
   // pull policy for the pod container

In other words: The Kubernetes runtime patches the pod with the final image when it is time to start that container. So, Always does in fact make Kubernetes pull the image on start!

That means that ImagePullPolicy: Always is effectively the same as pull: on_start, and is NOT the same as pull: always. If a step has pull: always and has a typo in the image or image tag, you won't find out about that until it is time for the image to run.

So, we need a way for the Kubernetes Runtime to query the Docker Registry to make sure the image exists for any steps/services/secrets with pull: always or pull: not_present. That check needs to happen at the start of the build before starting any other steps so that we can fail the build as soon as possible, similar to how the Docker runtime does it.

Value

Harmonize Docker and Kubernetes Runtime behaviors as far as reporting image problems earlier.

Definition of Done

Kubernetes runtime can

• query the docker registry for images with pull: always or pull: not_present at the beginning of builds
• return any missing image or missing image tag errors before starting any services, steps, or secrets.

Effort (Optional)

I've already figured out how to work with the docker libs to work with the docker registry.
See: https://gist.github.com/cognifloyd/effee107fc62ee4f722814ce94ee6d63

We will need to

• edit runtime/kubernetes/image.go: to allow CreateImage() to query the docker runtime.
• add CreateImage() calls in various places (following the pattern of the Docker runtime) to trigger the queries as needed.

We'll need to figure out

• how to store/load the additional docker-registry related config options (registry options, and registry auth options).
• can we reuse the image pull secrets from k8s somehow?

Impacted Personas (Optional)

Anyone who uses the Kubernetes runtime.

Make it easier for people to transition between runtimes as their behavior will be more similar.

[cognifloyd]

For docker registry credentials (authConfig in my gist), kubernetes has a kubernetes.io/dockerconfigjson type Secret that is exactly what we need.
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

At first glance, mounting that secret as a file in the worker pod seems like just the thing to do.
https://stackoverflow.com/questions/56361831/how-to-mount-a-kubernetes-io-dockerconfigjson

But, that only works if: (1) the worker is running in kubernetes, and (2) the pipeline pods are in the same namespace as the worker.
The second assumption is not true for me, and we explicitly design the worker so it can do its thing from outside of kubernetes.

So, what about using the kubernetes client (c.Kubernetes) to retrieve the secret from the target namespace.

Whatever ServiceAccount Vela is using would need permission to do so:

rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourcesNames: ["name-of-pull-secret"] // just the secret
  verbs: ["get"]
- apiGroups: [""]
  resources: ["serviceAcounts"]
  resourcesNames: ["name-of-service-account"] // just the service account.
  verbs: ["get"]

The kubernetes runtime should grab the secret that would be used to actually pull the image. So:

1. If the PipelinePodsTemplate has an ImagePullSecret, grab that
2. If the PipelinePodsTemplate specifies a ServiceAccount to use, then get that account's pull secret if any
3. If the default ServiceAccount has a pull secret, use that.
4. Try without authentication.

[cognifloyd]

From slack conversation with @JordanSussman (reordered for clarity)

I worry that the workers are setup with a registry mirror configuration (https://docs.docker.com/registry/recipes/mirror/#configure-the-docker-daemon). For example, all of our Vela workers utilize that configuration to avoid connecting to hub.docker.com directly.

I believe Podman has a similar configuration that allows mirroring any registry instead of just hub.docker.com.
https://github.com/containers/image/blob/main/docs/containers-registries.conf.5.md#remapping-and-mirroring-registries

I’m unsure how/if Kubernetes shows that info to the API or not.

There isn't a way to pull the registries config (including mirrors) from kubernetes. So, the Vela admin would also have to configure the Vela workers with those mirrors. That's one of my open questions: how to configure those bits.

---

containers/image looks like a sweet library. I wonder if it has a better interface than the docker libs (which are annoying)

I’ve also had good luck interacting with registries with https://github.com/google/go-containerregistry

google/go-containerregistry looks awesome and implements a lot of the kubernetes auth stuff and then some. That looks like a winner.

The only thing go-containerregistry doesn't seem to handle is mirrors. I don't see a way to configure mirrors that it should check first before pulling from the image's canonical registry

Stale feature request: google/go-containerregistry#1200

---

With k8s, I think I'll end up pulling the ImagePullSecret from kubernetes to communicate with the registry.

K8s ImagePullSecrets are just the docker cli's conf file.

This looks the same as the docker cli file: https://github.com/containers/image/blob/main/docs/containers-auth.json.5.md