imgpkg copy mismatches digests with images on quay.io

[cxhrouet]

What steps did you take:
I'm trying to relocate with imgpkg copy an image from quay.io registry to an internal private registry.

What happened:

imgpkg copy --lock sealed-secrets-v0.16.0.lock --to-repo private-registry/sealed-secrets-controller

with sealed-secrets-v0.16.0.lock being :

---
apiVersion: imgpkg.carvel.dev/v1alpha1
images:
- annotations:
    kbld.carvel.dev/id: quay.io/bitnami/sealed-secrets-controller:v0.16.0
  image: quay.io/bitnami/sealed-secrets-controller@sha256:cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37
kind: ImagesLock

resulted in exporting a wrong digest:

copy | exporting 1 images...
copy | will export quay.io/bitnami/sealed-secrets-controller@sha256:451a3e5dc6813d40a15af2e2e45d24e80782db54a8e7c3e9858c711e094da6b3
copy | exported 1 images
copy | importing 1 images...

 15.82 MiB / 15.82 MiB [===========================================================================================================================================================] 100.00% 88.62 MiB/s 0s

copy | done uploading images
Error: Expected image 'quay.io/bitnami/sealed-secrets-controller@sha256:cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37' to have been copied but was not

When exporting to tar, no error is thrown but the digest is still wrong :

copy | exporting 1 images...
copy | will export quay.io/bitnami/sealed-secrets-controller@sha256:451a3e5dc6813d40a15af2e2e45d24e80782db54a8e7c3e9858c711e094da6b3
copy | exported 1 images
copy | writing layers...
copy | done: file 'manifest.json' (7.245µs)
copy | done: file 'sha256-9ff2acc3204b4093126adab3fed72de8f7bbfe332255b199c30b8b185fcf6923.tar.gz' (621.310809ms)
copy | done: file 'sha256-6701582488b125cda944f349da838044ace43530fc9c385ec08684e809c45887.tar.gz' (3.349394772s)
Succeeded

with kbld relocate it seems to get the right digest but there is another error related to that hash (451a3e5dc6813d40a15af2e2e45d24e80782db54a8e7c3e9858c711e094da6b3) that I cannot find anywhere in the manifests :

relocate | exporting 1 images...
relocate | will export quay.io/bitnami/sealed-secrets-controller@sha256:cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37
relocate | exported 1 images
relocate | importing 1 images...
relocate | importing quay.io/bitnami/sealed-secrets-controller@sha256:cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37 -> private-registry/sealed-secrets-controller@sha256:cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37...
relocate | imported 0 images
kbld: Error: Importing image quay.io/bitnami/sealed-secrets-controller@sha256:cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37:
  Importing image index as private-registry/sealed-secrets-controller@sha256:cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37:
    Writing image index:
      Retried 5 times:
        PUT https://private-registry/v2/sealed-secrets-controller/manifests/kbld-sha256-cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37:
          BLOB_UNKNOWN: blob unknown to registry; sha256:451a3e5dc6813d40a15af2e2e45d24e80782db54a8e7c3e9858c711e094da6b3

What did you expect:
The proper image is copied on the target registry without issue.

Anything else you would like to add:

The copy from quay.iois working with skopeo without issue.

There is also no issue in relocating images from Docker HUB with the same imgpkg copy command.

Environment:

• imgpkg version (use imgpkg --version): 0.14.0
• Docker registry used (e.g. Docker HUB): quay.io
• OS (e.g. from /etc/os-release): Ubuntu 20.04.2

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[DennisDenuto]

Thanks @cxhrouet for raising this issue

There is a known issue around copying an image index which quay.io/bitnami/sealed-secrets-controller@sha256:cb183c914323dcad01a46e6c879368b2f4706a2c021a9386fb6a6a5e8768fa37 is

Can you try again with the latest version of imgpkg (which should have fixed this issue). https://github.com/vmware-tanzu/carvel-imgpkg/releases/tag/v0.15.0

[cxhrouet]

Indeed, it's working now with imgpkg v0.15.0 !