Retagging and pushing a multiarch manifest

[GunaKKIBM]

Scenario:

1. Let's say we pull an opensource image, for example registry.k8s.io/sig-storage/csi-provisioner:v3.4.1 . It is a manifest list as shown here.

gunakk@Gunas-MacBook-Pro Z-kube-config % docker manifest inspect registry.k8s.io/sig-storage/csi-provisioner:v3.4.1
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2261,
         "digest": "sha256:29c6b5ffc0f371249e4e165e1108b4e57d5da17c411c9bd652370610676e5474",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2261,
         "digest": "sha256:d3a188fd5f7cc129b650c7ce0f4958a08ff3102ee696879341a64e3d835330b3",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2261,
         "digest": "sha256:926a9a3cc1fd34633d844ba09a8ccbdfc11b7107834ebb2dbda4e1a74644ffc8",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2261,
         "digest": "sha256:f64997c5a803833d727a6602bf313aa1968bd234252200f044d9315ccfe4ad84",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2261,
         "digest": "sha256:33514ef1c1e914f229b51a970a1301625dd10aad0981542a676283ca69f230b1",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2261,
         "digest": "sha256:bc9259023553bf5b8bd6fe5dbbd098269f98d9dd474b3fc2e624f1de06010dda",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

2. If we retag the image, it doesn't get retagged as manifest list, rather a docker image with local architecture

gunakk@Gunas-MacBook-Pro Z-kube-config % docker tag registry.k8s.io/sig-storage/csi-provisioner:v3.4.1 gunakk777/local-csi-provisioner:exp
gunakk@Gunas-MacBook-Pro Z-kube-config % 
gunakk@Gunas-MacBook-Pro Z-kube-config % 
gunakk@Gunas-MacBook-Pro Z-kube-config % docker manifest inspect gunakk777/local-csi-provisioner:exp
no such manifest: docker.io/gunakk777/local-csi-provisioner:exp
gunakk@Gunas-MacBook-Pro Z-kube-config % 
gunakk@Gunas-MacBook-Pro Z-kube-config % docker inspect gunakk777/local-csi-provisioner:exp
[
    {
        "Id": "sha256:8cab7ddf9bed2158caedfde3728f05ef97c46e2537a074f80552f7ad102d111d",
        "RepoTags": [
            "gunakk777/csi-provisioner:v3.4.1",
            "gunakk777/local-csi-provisioner:exp",
            "local-csi-provisioner:exp",
            "registry.k8s.io/sig-storage/csi-provisioner:v3.4.1"
        ],
        "RepoDigests": [
            "registry.k8s.io/sig-storage/csi-provisioner@sha256:893e37a388e7a7463d6c3523311b28cfbc5ae536dbef35430eed272cdc6850dc"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2023-04-05T18:26:07.02790064Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "0",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
            ],
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "/csi-provisioner"
            ],
            "OnBuild": null,
            "Labels": {
                "description": "CSI External Provisioner",
                "maintainers": "Kubernetes Authors",
                "revision": "v3.4.1"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 64142265,
        "VirtualSize": 64142265,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/eb993127b3d04e0e4418be4dadc207cf5bf4066d7691bbc0bbcf64d655b23586/diff:/var/lib/docker/overlay2/5983de85fe7fbf7488f797ff2e4545af689ec44318e8236236c700b8814090cc/diff:/var/lib/docker/overlay2/caf950d3ef9dd02c1dff8c11c862143ddd4e722892befe0ab75681503cd24ead/diff:/var/lib/docker/overlay2/1de461f7f924098ea72b8be151b78ea3e331718071741ef9871db47fe3936e65/diff:/var/lib/docker/overlay2/87ae008e2d90862df88c0bca097c4e4523339681b57d01980f10b0df1cb8f0fd/diff:/var/lib/docker/overlay2/b688a6208c4db777065965a608a1e508ca403722e40ba8c82d31852d07660225/diff:/var/lib/docker/overlay2/7da00e289cd27a1558a32878161e836b29db637a674a62083e209cb41b9950de/diff:/var/lib/docker/overlay2/ef7d86421a197402f52e34b93641363c05362d9d66714b7b39d2974c47d191e7/diff:/var/lib/docker/overlay2/02dc743ad8d6d36f372dfb8ca3eec51e40379176cad8a89d13f2630a75403c35/diff",
                "MergedDir": "/var/lib/docker/overlay2/923243df1ddc3f82e432d98bd35e9a1de07b7785e323131648fd5473fe840638/merged",
                "UpperDir": "/var/lib/docker/overlay2/923243df1ddc3f82e432d98bd35e9a1de07b7785e323131648fd5473fe840638/diff",
                "WorkDir": "/var/lib/docker/overlay2/923243df1ddc3f82e432d98bd35e9a1de07b7785e323131648fd5473fe840638/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:d0157aa0c95a4cae128dab97d699b2f303c8bea46914dc4a40722411f50bb40e",
                "sha256:6fbdf253bbc2490dcfede5bdb58ca0db63ee8aff565f6ea9f918f3bce9e2d5aa",
                "sha256:399826b51fcf6c959b7a7e86b89ac1ee6685d64da54e5223e1d182c491a1bbd6",
                "sha256:ff5700ec54186528cbae40f54c24b1a34fb7c01527beaa1232868c16e2353f52",
                "sha256:d52f02c6501c9c4410568f0bf6ff30d30d8290f57794c308fe36ea78393afac2",
                "sha256:e624a5370eca2b8266e74d179326e2a8767d361db14d13edd9fb57e408731784",
                "sha256:1a73b54f556b477f0a8b939d13c504a3b4f4db71f7a09c63afbc10acb3de5849",
                "sha256:d2d7ec0f6756eb51cf1602c6f8ac4dd811d3d052661142e0110357bf0b581457",
                "sha256:4cb10dd2545bd173858450b80853b850e49608260f1a0789e0d0b39edf12f500",
                "sha256:29d8a6b06ea1f0c984188d49f0411f66bc88469df9bf993559acd9a9529d5362"
            ]
        },
        "Metadata": {
            "LastTagTime": "2023-04-18T07:16:30.331550482Z"
        }
    }
]

3. I have a requirement where, I need to push this manifest to my docker hub. One way to do it is using the following commands

Pull for different architectures
echo "FROM --platform=linux/<arch> <image>:<tag>" | docker build --pull -t <our-own-registry:tag> -

echo "FROM --platform=linux/<arch> <image>:<tag>" | docker build --pull -t <our-own-registry:tag> -

Create manifest

docker manifest create <image:tag>  <our-own-registry:tag>  <our-own-registry:tag>

Wouldn't it be better, if we can pull the manifest, have a way to tag it and push the same directly. Or is there any way that exists?

Thank you!

[jedevc]

Yup, this functionality exists today, you can use docker buildx imagetools create, see here.

Unfortunately, only a single architecture image can be pulled locally, though there is ongoing work to improve that.

[thiagotgo90]

What worked for me was:

docker buildx imagetools inspect --raw <IMAGE>:<TAG> | jq '.manifests[] |select(.platform.architecture |contains("arm"))' > descriptor_arm.json
docker buildx imagetools inspect --raw <IMAGE>:<TAG> | jq '.manifests[] |select(.platform.architecture |contains("amd"))' > descriptor_amd.json
docker buildx imagetools create -t <REPOSITORY>/<IMAGE>:<TAG> -f descriptor_amd.json -f descriptor_arm.json

My original image was published using docker buildx build. Here is the inspect output:

docker buildx imagetools inspect --raw <registry>/<image>:<tag> | jq                                                              
{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:<arm sha>",
      "size": 2953,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:<amd sh>",
      "size": 2953,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:XXXX",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:<arm sha>",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:YYYYY",
      "size": 566,
      "annotations": {
        "vnd.docker.reference.digest": "sha256:<amd sh>",
        "vnd.docker.reference.type": "attestation-manifest"
      },
      "platform": {
        "architecture": "unknown",
        "os": "unknown"
      }
    }
  ]
}

That published a new image index type into my registry pointing to the two existing docker image in different archs.

Obs: I have not added those two last manifests with annotations, maybe I need to, but I ran a test without them and it worked.

Hope that helps you.

[puchalam-devops]

hi Guys
thats very helpful, cheers

[GunaKKIBM]

This was helpful, thank you!

[thomaseizinger]

We managed to solve this with the following command:

docker buildx imagetools create -t example-registry1.com/image:tag example-registry2.com/image:tag

This will copy example-registry2.com/image:tag to example-registry1.com/image:tag, even for multiplatform images. Credit to @jamilbk!

[thamerbelfkihthamer]

very helpful, thank you guys 🥳

[elamaran11]

@thomaseizinger Does this work if both the registries are private for multi arch images?

[jamilbk]

@elamaran11 Assuming you've authenticated to them, yes it should. Or at least it is for us.

[elamaran11]

@jamilbk The moment we log in to second ECR, we are getting errors.