Add support for OCI images #717

sjdaws · 2023-04-11T05:57:06Z

Update all dependencies
Update to kube v1.26.3
Add support for OCI images (fixes fails to pull images built for single platform with docker build and push action #715)

Previous error with OCI images:

time="2023-04-10T03:34:33Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get \"https://index.docker.io/v2/vaultwarden/server/manifests/1.25.1\": http: non-successful response (status=404 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"MANIFEST_UNKNOWN\\\",\\\"message\\\":\\\"OCI index found, but accept header does not support OCI indexes\\\"}]}\\n\")" image="vaultwarden/server:1.25.1" password= username=
time="2023-04-10T03:34:33Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get \"https://index.docker.io/v2/vaultwarden/server/manifests/1.25.1\": http: non-successful response (status=404 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"MANIFEST_UNKNOWN\\\",\\\"message\\\":\\\"OCI index found, but accept header does not support OCI indexes\\\"}]}\\n\")" image="namespace:misc,image:index.docker.io/vaultwarden/server:1.25.1,provider:kubernetes,trigger:poll,sched:@every 15m,secrets:[]"
time="2023-04-10T03:34:33Z" level=error msg="trigger.poll.manager: got error(-s) while watching images" error="encountered errors while adding images: Get \"https://index.docker.io/v2/vaultwarden/server/manifests/1.25.1\": http: non-successful response (status=404 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"MANIFEST_UNKNOWN\\\",\\\"message\\\":\\\"OCI index found, but accept header does not support OCI indexes\\\"}]}\\n\")"

After fix:

time="2023-04-11T05:47:54Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:dd8cf61d1997c098cc5686ef3116ca5cfef36f12192c01caa1de79a968397d4c" image="vaultwarden/server:1.25.1" job_name=index.docker.io/vaultwarden/server schedule="@every 2s"
time="2023-04-11T05:47:56Z" level=warning msg="provider.kubernetes: got error while archiving approvals counter after successful update" error="approval not found: record not found" kind=deployment name=deployment-3 namespace=keel-e2e-test-5j6fs
time="2023-04-11T05:47:56Z" level=info msg="provider.kubernetes: resource updated" kind=deployment name=deployment-3 namespace=keel-e2e-test-5j6fs new=1.25.2 previous=1.25.1

krzwiatrzyk · 2023-04-11T17:58:48Z

@rusenask can you look at this PR? It seems to be very well prepared and solves an issue that makes Keel unusable.

rusenask · 2023-04-11T19:09:41Z

looks good!

travisghansen · 2023-04-11T21:35:00Z

Let’s get this in a release!

travisghansen · 2023-04-15T00:54:04Z

I just tried imageID: docker.io/keelhq/keel@sha256:fafbe949fb7d3cec309f859b62ba541b81aa17415f7be6290c7ea07f467ecb6c and I'm still getting copious amounts of OCI manifest found, but accept header does not support OCI manifests messages.

Should I expect this to be resolved in that image tag?

sjdaws · 2023-04-15T00:55:37Z

In the interim, until there is an official keel release, I am using:

helm upgrade --install keel keel/keel \
    --set image.repository="sjdaws/keel" \
    --set image.tag="0.18.1"

travisghansen · 2023-04-15T01:04:28Z

Thanks for knocking this feature out!

Just tried that image (imageID: docker.io/sjdaws/keel@sha256:9409a3309b33a78cfb4086a81e84e833b87d8c4cd47f120edacce4acdc4ad9d6) and still getting the errors :( For what it's worth the registry is gitlab's (self-hosted) registry using auth (logs do show username/password being used).

sjdaws · 2023-04-15T01:07:19Z

Happy to test if you can find a public gitlab repo that uses OCI images.

travisghansen · 2023-04-15T02:13:10Z

I suspect it has less to do with gitlab and maybe more about manifest? I'm using buildah to create the images and perhaps something is slightly different than other?

skopeo inspect --raw docker://<imag>:latest | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:3f57168dbb7ad396136fb587e62c1e6b420e988c3d69a9748193e12d38112452",
    "size": 23747
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:2ef0edd75722783cd9f84285a37372e8521f2b49961bf6087a24d0de4b59658d",
      "size": 75546229
    },
...

Are you sending application/vnd.oci.image.index.v1+json or application/vnd.oci.image.manifest.v1+json?

Could be similar to this: https://bugs.launchpad.net/cloud-images/+bug/2007408

sjdaws · 2023-04-15T03:24:30Z

https://github.com/keel-hq/keel/blob/master/registry/docker/manifest.go#L28 relates to https://github.com/opencontainers/image-spec/blob/main/specs-go/v1/mediatype.go#L28.

This is the test for docker, you'll need to get a bearer token for your repo from gitlab:

export DOCKER_REPO=vaultwarden/server; 
export DOCKER_TOKEN=$(curl --silent "https://auth.docker.io/token?scope=repository:$DOCKER_REPO:pull&service=registry.docker.io" | jq -r '.token'); \
    curl \
        --silent \
        --header "Accept: application/vnd.oci.image.index.v1+json" \
        --header "Authorization: Bearer $DOCKER_TOKEN" \
        "https://index.docker.io/v2/$DOCKER_REPO/manifests/latest"

Can always add another fallback to manifest header if required.

travisghansen · 2023-04-15T04:05:59Z

So it seems the issue is when the image isn't a multi-arch image?

Should we not be sending --header "Accept: application/vnd.oci.image.index.v1+json, application/vnd.oci.image.manifest.v1+json" along with application/vnd.docker.distribution.manifest.v2+json and whatever other non-oci values should be sent?

sjdaws · 2023-04-15T04:08:47Z

To reiterate my previous comment:

Happy to test if you can find a public gitlab repo that uses OCI images.

Give me an image and I'll get it working, the current code works for all of the images I use.

travisghansen · 2023-04-15T04:11:12Z

Let me see if I can find one somewhere that's not multi arch real quick (that's public)...

travisghansen · 2023-04-15T04:25:12Z

OK, so I think you can send all 3 values in the same request FYI to reduce the number of requests.

skopeo inspect --raw docker://travisghansen/playground:alpine-buildah | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:f67898f28f024eefca03968d48f47f820cf0cb0d2d4b361e3bcff156b4ebfda6",
    "size": 834
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f56be85fc22e46face30e2c3de3f7fe7c15f8fd7c4e5add29d7f64b87abdaa09",
      "size": 3374563
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a22f44112e8c584817b6512921e613b750f51cb703bc2419220deaf55f914304",
      "size": 256
    }
  ],
  "annotations": {
    "org.opencontainers.image.base.digest": "sha256:b6ca290b6b4cdcca5b3db3ffa338ee0285c11744b4a6abaa9627746ee3291d8d",
    "org.opencontainers.image.base.name": "docker.io/library/alpine:latest"
  }
}

sjdaws · 2023-04-15T04:28:08Z

Give this image a try, if it works I'll raise a PR: sjdaws/keel:test - sha256:466af8c915b24c7096a6274a8575b1ddc6d5597e644b0562445e245e1fdd0cf5

I've added the manifest header and condensed them into a single call as suggested.

travisghansen · 2023-04-15T04:57:58Z

Yes!

time="2023-04-15T04:46:04Z" level=info msg="trigger.poll.RepositoryWatcher: new watch tag digest job added" digest="sha256:85c6c683436bf20da51b24b760974c03f13cc79e8207d544ac34eade94a6e278" image="travisghansen/playground:alpine-buildah" job_name="index.docker.io/travisghansen/playground:alpine-buildah" schedule="@every 1m"
time="2023-04-15T04:49:04Z" level=info msg="trigger.poll.WatchTagJob: digest change detected, submiting event to providers" image="travisghansen/playground:alpine-buildah" new_digest="sha256:1d3a83f0b0ebef9e501f930c9ccf100f8053c3d9d1b9ebe9281a702a17f6696f"
time="2023-04-15T04:49:04Z" level=warning msg="provider.kubernetes: got error while archiving approvals counter after successful update" error="approval not found: record not found" kind=deployment name=nginx-deployment namespace=default
time="2023-04-15T04:49:05Z" level=info msg="provider.kubernetes: resource updated" kind=deployment name=nginx-deployment namespace=default new=alpine-buildah previous=alpine-buildah

Another suggestion, I think GET requests count against rate limits on docker.io. We should be using HEAD here: https://github.com/keel-hq/keel/blob/master/registry/docker/manifest.go#L18

As it should help with rate-limiting and decrease (in some cases significantly) the amount of data/bandwidth.

For example:

export DOCKER_REPO=vaultwarden/server; 
export DOCKER_TOKEN=$(curl --silent "https://auth.docker.io/token?scope=repository:$DOCKER_REPO:pull&service=registry.docker.io" | jq -r '.token'); \
    curl \
        --head \
        --verbose \
        --header "Accept: application/vnd.oci.image.index.v1+json" \
        --header "Authorization: Bearer $DOCKER_TOKEN" \
        "https://index.docker.io/v2/$DOCKER_REPO/manifests/latest"

*   Trying 34.194.164.123:443...
* Connected to index.docker.io (34.194.164.123) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=*.docker.com
*  start date: Feb 22 00:00:00 2023 GMT
*  expire date: Nov 30 23:59:59 2023 GMT
*  subjectAltName: host "index.docker.io" matched cert's "*.docker.io"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
* using HTTP/1.x
> HEAD /v2/vaultwarden/server/manifests/latest HTTP/1.1
> Host: index.docker.io
> User-Agent: curl/8.0.1
> Accept: application/vnd.oci.image.index.v1+json
> Authorization: Bearer REDACTED> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< content-length: 3161
content-length: 3161
< content-type: application/vnd.oci.image.index.v1+json
content-type: application/vnd.oci.image.index.v1+json
< docker-content-digest: sha256:c56cba7d646584e73a17604f4d4e5aba95ee4198bbed6c919c9514d2ada97d04
docker-content-digest: sha256:c56cba7d646584e73a17604f4d4e5aba95ee4198bbed6c919c9514d2ada97d04
< docker-distribution-api-version: registry/2.0
docker-distribution-api-version: registry/2.0
< etag: "sha256:c56cba7d646584e73a17604f4d4e5aba95ee4198bbed6c919c9514d2ada97d04"
etag: "sha256:c56cba7d646584e73a17604f4d4e5aba95ee4198bbed6c919c9514d2ada97d04"
< date: Sat, 15 Apr 2023 04:53:34 GMT
date: Sat, 15 Apr 2023 04:53:34 GMT
< strict-transport-security: max-age=31536000
strict-transport-security: max-age=31536000
< docker-ratelimit-source: REDACTED
docker-ratelimit-source: REDACTED

sjdaws · 2023-04-15T05:07:07Z

HEAD calls don't return the correct docker-content-digest for all images.

e.g.

export DOCKER_REPO=vaultwarden/server;
export DOCKER_TOKEN=$(curl --silent "https://auth.docker.io/token?scope=repository:$DOCKER_REPO:pull&service=registry.docker.io" | jq -r '.token'); \
    curl \
        --head \
        --verbose \
        --header "Accept: application/vnd.oci.image.index.v1+json" \
        --header "Authorization: Bearer $DOCKER_TOKEN" \
        "https://index.docker.io/v2/$DOCKER_REPO/manifests/1.25.1"

returns docker-content-digest: sha256:def922bcdfae8e796f46dd83c11c2aaa8fb4fd8eaa3a81428efe0c3fc4bab8b4 but the actual digest is sha256:dd8cf61d1997c098cc5686ef3116ca5cfef36f12192c01caa1de79a968397d4c. This is a known issue that has a complex workaround involving multiple calls depending on the image type.

If you can get it working with the current tests passing raise a PR.

PR for this fix raised here: #718

travisghansen · 2023-04-15T05:21:35Z

Wow! Very interesting little tidbit. Do you know how prevalent that is?

When you execute that command as a GET it returns no such header? (Sorry, afk now)

sjdaws · 2023-04-15T05:27:09Z

Not 100% sure, I only came across that issue because I tried HEAD initially and some of the tests started failing.

travisghansen · 2023-04-15T13:57:57Z

Just doing a bit more research on the HEAD issue. Can you send over which tests were failing?

Even with oci index/manifest responses the etag/docker-content-digest is not the sha of any of the layers. My golang skills are quite lacking but reading the code seem to make the point irrelevant anyway as if the header is there that is what gets returned (and it's there in the example cited, and it's the same with both GET and HEAD). Am I missing something?

travisghansen · 2023-04-15T14:14:39Z

As an FYI, without application/vnd.docker.distribution.manifest.v2+json in the accept header the value returned is:

docker-content-digest: sha256:def922bcdfae8e796f46dd83c11c2aaa8fb4fd8eaa3a81428efe0c3fc4bab8b4

with it (what this code does now) the value returned is:

docker-content-digest: sha256:dd8cf61d1997c098cc5686ef3116ca5cfef36f12192c01caa1de79a968397d4c

travisghansen · 2023-04-20T14:21:09Z

@sjdaws your build seems to be working well FYI for buildah images. Any further thoughts on the HEAD discussion?

sjdaws · 2023-04-20T21:23:05Z

@travisghansen I've done a test here: #721

With HEAD only, it fails for insecure registries: https://drone-kr.webrelay.io/keel-hq/keel/455/1/2
With HEAD first, then fall back to GET and read body it fails on get digest, although it seems to use HEAD most of the time: https://drone-kr.webrelay.io/keel-hq/keel/456/1/2. There is a single fallback ("HEAD request failed, attempting to fetch digest from body") for the insecure registry test.
With HEAD first, then fall back to GET, while still parsing the header even on GET, it fails on get digest: https://drone-kr.webrelay.io/keel-hq/keel/457/1/2, still with a single fall back for insecure registries.

I'll look into this a bit more, but all these tests need to pass before the change can be made.

travisghansen · 2023-04-20T21:29:54Z

Awesome! Are the tests that fail executed locally by simply running make test? I'd like to see what I can help with locally..

sjdaws · 2023-04-20T21:31:29Z

The pass when running go test -v ./registry but fail with the CI, so there's something going on with the CI. It looks like the URL is passed twice:

registry.manifest.head url=http://127.0.0.1:39913/v2/http://127.0.0.1:39913/manifests/notimportant repository=http://127.0.0.1:39913 reference=notimportant

http://127.0.0.1:39913/v2/http://127.0.0.1:39913

sjdaws · 2023-04-20T21:49:44Z

It's actually failing locally too, go test -v ./registry/docker, so I'll debug it locally which is much easier.

sjdaws · 2023-04-20T22:42:28Z

Ok, it's up and running here: #721.

Testing shows it only falls back to insecure registry which is a test registry that purposely doesn't have a docker-content-digest header.

sjdaws added 5 commits April 11, 2023 11:51

Update all depedencies, shift to kube v1.26.3

a58d5a6

Add support for OCI images

2315ace

Fix grammar

bc5a4cb

Fix tag pagination

ed45f87

Fix tag pagination

1e76c36

rusenask merged commit da098d8 into keel-hq:master Apr 11, 2023

sjdaws deleted the fix-oci-images branch April 11, 2023 21:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for OCI images #717

Add support for OCI images #717

sjdaws commented Apr 11, 2023 •

edited

Loading

krzwiatrzyk commented Apr 11, 2023

rusenask commented Apr 11, 2023

travisghansen commented Apr 11, 2023

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023 •

edited

Loading

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023 •

edited

Loading

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023 •

edited

Loading

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023

travisghansen commented Apr 15, 2023

travisghansen commented Apr 20, 2023

sjdaws commented Apr 20, 2023 •

edited

Loading

travisghansen commented Apr 20, 2023

sjdaws commented Apr 20, 2023

sjdaws commented Apr 20, 2023

sjdaws commented Apr 20, 2023

Add support for OCI images #717

Add support for OCI images #717

Conversation

sjdaws commented Apr 11, 2023 • edited Loading

krzwiatrzyk commented Apr 11, 2023

rusenask commented Apr 11, 2023

travisghansen commented Apr 11, 2023

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023 • edited Loading

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023 • edited Loading

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023 • edited Loading

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023

sjdaws commented Apr 15, 2023

travisghansen commented Apr 15, 2023

travisghansen commented Apr 15, 2023

travisghansen commented Apr 20, 2023

sjdaws commented Apr 20, 2023 • edited Loading

travisghansen commented Apr 20, 2023

sjdaws commented Apr 20, 2023

sjdaws commented Apr 20, 2023

sjdaws commented Apr 20, 2023

sjdaws commented Apr 11, 2023 •

edited

Loading

travisghansen commented Apr 15, 2023 •

edited

Loading

travisghansen commented Apr 15, 2023 •

edited

Loading

sjdaws commented Apr 15, 2023 •

edited

Loading

sjdaws commented Apr 20, 2023 •

edited

Loading