Jib should check and refresh access tokens as required #691
Comments
This comment has been minimized.
This comment has been minimized.
Does this mean if the build succeeds it takes a lot less than 8-10 minutes? Do you remember how long it took when it succeeded? |
This comment has been minimized.
This comment has been minimized.
|
Hmm... could it be that the credential we retrieve expires after a few minutes? However, this is just a completely wild guess. |
This comment has been minimized.
This comment has been minimized.
|
@ghostbuster91 is this still happening with |
I think my suspicison was right. I can reproduce this locally with Docker Hub. In The result is the 401 Unauthorized. Thread.sleep(3 * 60 * 1000);
t2.lap("pushBlob PATCH " + blobDigest);
// PATCH <Location> with BLOB
URL putLocation = callRegistryEndpoint(blobPusher.writer(patchLocation));
Preconditions.checkNotNull(putLocation);
Thread.sleep(5 * 60 * 1000);
t2.lap("pushBlob PUT " + blobDigest);
// PUT <Location>?digest={blob.digest}
callRegistryEndpoint(blobPusher.committer(putLocation));Output: My previous analysis that I didn't post as a comment, as I wasn't sure: Analysis (click the arrow to expand)
|
|
BTW, to reproduce this continuously, you'll need to keep modifying your test project to create fresh new blob digests, since it looks like blobs pushed to Docker Hub are retained at least for some time even if you delete the image on Docker Hub. |
|
@ghostbuster91 one possible workaround is to do |
|
Enabling tracing, I see the docker OAuth response provides a 300s timeout on the access-token: So we fetch the authorization once and then use it for all subsequent operations. We should instead pull the auth and check for each operation. |
|
@chanseokoh @briandealwis Nice finds! Yea, we should re-authenticate a token if a request fails with unauthorized. |
|
Hi @ghostbuster91 , just checking back in to see if this is still a problem you are experiencing? Does anyone else experience this issue? |
This comment has been minimized.
This comment has been minimized.
|
im now facing the same kind of problem.is there any way to get rid off it |
|
Hi @danimeko, If you are facing the same problem where it takes a very long time to push a large base image and your credentials expire during the long push, do #691 (comment) once and leave at least one image in your target repository. That should make Jib skip pushing the base image layers because they already exist in your repository. |
|
Another point: Jib 1.4.0 now supports cross-repository mounts which will speed up pushing images to the same registry (e.g., from Docker Hub and to Docker Hub). |
|
thank you for responses. ./gradlew jibDockerBuild need docker daemon running which is not allowed in our system. That is why we use ./gradlew jib |
|
Then you can just push the base image (no Jib involved) to the target repository by any means, like running skopeo on your laptop which doesn't need Docker. |
briandealwis
changed the title
|
FYI @iocanel |
|
@ghostbuster91 @danimeko @nithinkota @iocanel we've released 2.0.0 with the fix. Jib will now automatically refresh expired bearer auth tokens. This fixes build failures with "401 Unauthorized", for example, when pushing to Docker Hub takes more than 5 minutes due to slow network. |
|
Awesome, thank you very much for this great work! |
Description of the issue:
Hi, I don't know why this happens but when I specify a larger image as base image the build fails.
Both images are in the same repository. The only difference is the size, smaller is something around 50 MB and bigger 500 MB. Maybe that's just a problem with my network, but again it is hard to tell based on the error message. Actually when I look at docker.hub interface, everything looks ok, I mean that the new image was uploaded.
Expected behavior:
Sometimes build passes but very rarely, so that would be the expected behavior.
Steps to reproduce:
It is hard for me to provide some steps since both images are in a private repository.
Environment:
Ubuntu 18.04, jdk 1.8
jib-gradle-pluginConfiguration:plugins { id 'com.google.cloud.tools.jib' version '0.9.6' } apply plugin: 'java' group 'test-jib' version '1.0-SNAPSHOT' sourceCompatibility = 1.8 repositories { mavenCentral() } dependencies { testCompile group: 'junit', name: 'junit', version: '4.12' } jib { from { image = 'userfeeds/base-java-image' } to { image = 'ghostbuster91/test-jib' } container { mainClass = 'Main' } }Log output:
Additional Information:
What is common for this failures is that they always took around 8-10 minutes.
The text was updated successfully, but these errors were encountered: