imgpkg v0.18.0 auth struggles with docker's credsStore feature

[dhadukk]

What steps did you take:

1. docker login dev.registry.tanzu.vmware.com
2. imgpkg copy --tar developer-conventions-bundle-v0.1.0-alpha.tar --to-repo dev.registry.tanzu.vmware.com/tanzu-advanced-edition/developer-conventions
   What happened:
   copy | Error: Error uploading images: HEAD https://dev.registry.tanzu.vmware.com/v2/tanzu-advanced-edition/developer-conventions/blobs/sha256:214eb19e7b1a02c2520a4858ceb6651ecf58ca6a6a08d94d6613571d9a538cdf: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
   copy | Error: Error uploading images: HEAD https://dev.registry.tanzu.vmware.com/v2/tanzu-advanced-edition/developer-conventions/blobs/sha256:c6e693ed03f16e99181e779d1766e7c2ff0bcfc8ad7a21fed5f17587c4e98696: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
   copy | Error: Error uploading images: HEAD https://dev.registry.tanzu.vmware.com/v2/tanzu-advanced-edition/developer-conventions/blobs/sha256:214eb19e7b1a02c2520a4858ceb6651ecf58ca6a6a08d94d6613571d9a538cdf: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
   copy | Error: Error uploading images: HEAD https://dev.registry.tanzu.vmware.com/v2/tanzu-advanced-edition/developer-conventions/blobs/sha256:72164b581b02b1eb297b403bcc8fc1bfa245cb52e103a3a525a0835a58ff58e2: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
   copy | Error: Error uploading images: HEAD https://dev.registry.tanzu.vmware.com/v2/tanzu-advanced-edition/developer-conventions/blobs/sha256:0d7d70899875b098ace120b574f57b39e91415ceaf9b348d6abe168537509f5a: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)
   0 B / ? [-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------=] 0.00% 2562047h47m16s

copy | done uploading images
imgpkg: Error: Retried 5 times:
HEAD https://dev.registry.tanzu.vmware.com/v2/tanzu-advanced-edition/developer-conventions/blobs/sha256:0d7d70899875b098ace120b574f57b39e91415ceaf9b348d6abe168537509f5a: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)

What did you expect:
imgpkg copy successful.

Anything else you would like to add:
it works fine with imgpkg 0.17.

Environment:

• imgpkg version (use imgpkg --version): 0.18
• Docker registry used (e.g. Docker HUB): dev.registry.tanzu.vmware.com, harbor-repo.vmware.com, Azure container registry)
• OS (e.g. from /etc/os-release): Mac

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[DennisDenuto]

Hi @dhadukk thanks for raising this bug

Are you running the imgpkg copy command from a VM / container running on an IaaS (GCP, AWS, Azure?). If so, could you try running the command with the following env var export IMGPKG_ENABLE_IAAS_AUTH=false

The thought here is that imgpkg is getting the cred from the IaaS metadata service and not from your dockercfg file via docker login

If the above doesn't apply / work:

• Can you try again but setting your credentials via the env variables and let us know if that fixes it for you?

export IMGPKG_REGISTRY_HOSTNAME=dev.registry.tanzu.vmware.com 
export IMGPKG_REGISTRY_USERNAME=foo
export IMGPKG_REGISTRY_PASSWORD=bar

• If you can also provide your docker cli version you are using to docker login that could be useful

[cppforlife]

root cause was identified to be iaas related auth handler starts picking up dockerconfig but is not able to work with more modern credsStore configuration.

current suggested workaround is to keep using v0.17.0 or use export IMGPKG_ENABLE_IAAS_AUTH=false. we'll be cutting a new release with a fix shortly.

[DennisDenuto]

A fix has been included in 0.19.0 https://github.com/vmware-tanzu/carvel-imgpkg/releases/tag/v0.19.0