Failed to push blob to local repo, error: http status code: 401

[devulm]

hi details about the setup, using 2.5.1 version on OpenShift 4.11,

harbor-core:v2.5.1

So have configured proxy cache projects for upstream registries, when trying to pull images , we intermittently see the below message and we dont see the repo created in the UI as well, the folders are visible when I check the volume.

Unsuccessful
Failed to push blob to local repo, error: http status code: 401, body: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to access repository:

successful

2023-06-28T09:51:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'InternalArtifact' on topic 'PUSH_ARTIFACT': ID-5, Repository-prod.registry.docker.site.gs.com/goharbor/harbor-exporter Tags-[v2.5.1] Digest-sha256:566be56fa88b377d858ec846832cef26542dfea1e5001b7986f9c22f3e6907b1 Operator-harbor#proxy-cache-service OccurAt-2023-06-28 09:51:51
2023-06-28T09:51:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'InternalArtifact' on topic 'PULL_ARTIFACT': ID-5, Repository-prod.registry.docker.site.gs.com/goharbor/harbor-exporter Tags-[v2.5.1] Digest-sha256:566be56fa88b377d858ec846832cef26542dfea1e5001b7986f9c22f3e6907b1 Operator- OccurAt-2023-06-28 09:51:51
2023-06-28T09:51:51Z [INFO] [/controller/event/handler/webhook/artifact/artifact.go:75]: []
2023-06-28T09:51:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ArtifactWebhook' on topic 'PUSH_ARTIFACT': ID-5, Repository-prod.registry.docker.site.gs.com/goharbor/harbor-exporter Tags-[v2.5.1] Digest-sha256:566be56fa88b377d858ec846832cef26542dfea1e5001b7986f9c22f3e6907b1 Operator-harbor#proxy-cache-service OccurAt-2023-06-28 09:51:51
2023-06-28T09:51:51Z [INFO] [/controller/event/handler/webhook/artifact/artifact.go:75]: []
2023-06-28T09:51:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ArtifactWebhook' on topic 'PULL_ARTIFACT': ID-5, Repository-prod.registry.docker.site.gs.com/goharbor/harbor-exporter Tags-[v2.5.1] Digest-sha256:566be56fa88b377d858ec846832cef26542dfea1e5001b7986f9c22f3e6907b1 Operator- OccurAt-2023-06-28 09:51:51
2023-06-28T09:51:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'AuditLog' on topic 'PULL_ARTIFACT': ID-5, Repository-prod.registry.docker.site.gs.com/goharbor/harbor-exporter Tags-[v2.5.1] Digest-sha256:566be56fa88b377d858ec846832cef26542dfea1e5001b7986f9c22f3e6907b1 Operator- OccurAt-2023-06-28 09:51:51
2023-06-28T09:51:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'Replication' on topic 'PUSH_ARTIFACT': ID-5, Repository-prod.registry.docker.site.gs.com/goharbor/harbor-exporter Tags-[v2.5.1] Digest-sha256:566be56fa88b377d858ec846832cef26542dfea1e5001b7986f9c22f3e6907b1 Operator-harbor#proxy-cache-service OccurAt-2023-06-28 09:51:51
2023-06-28T09:51:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'AuditLog' on topic 'PUSH_ARTIFACT': ID-5, Repository-prod.registry.docker.site.gs.com/goharbor/harbor-exporter Tags-[v2.5.1] Digest-sha256:566be56fa88b377d858ec846832cef26542dfea1e5001b7986f9c22f3e6907b1 Operator-harbor#proxy-cache-service OccurAt-2023-06-28 09:51:51
2023-06-28T09:51:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'P2PPreheat' on topic 'PUSH_ARTIFACT': ID-5, Repository-prod.registry.docker.site.gs.com/goharbor/harbor-exporter Tags-[v2.5.1] Digest-sha256:566be56fa88b377d858ec846832cef26542dfea1e5001b7986f9c22f3e6907b1 Operator-harbor#proxy-cache-service OccurAt-2023-06-28 09:51:51

[stonezdj]

How many harbor-core pod in your env?

[devulm]

we have 3 replicas each for harbor-core , harbor-registry and harbor-portal.

[cjvirtucio87]

I'm running into the same issue with 2.8.1 using the installer. probably also worth noting that I have it running behind nginx, but I use SSL pass-through, unlike this comment. I'm able to docker login and also log into the web UI of the registry. setting relativeurls in the config YAML file didn't work either.

[devulm]

Hi Wanted to check if any update on this ?

[ugurmendi]

we have 3 replicas each for harbor-core , harbor-registry and harbor-portal.

Hi devulm for the replication of the core there was an error about bearer token has not set because of the difference between time difference of the replicas when I was searching in the forums setting NTP might help you to solve problem

[devulm]

@ugurmendi these are all OpenShift nodes and have chrony configured, and sync with the same source, but using 1 replica does work and dont see the unauthorized messages but having only one replica will cause outages during cluster maintenance, is there any config that needs to be set for this to work with multiple replicas ?

Also if you were referring to this issue goharbor/harbor-helm#1205

we dont see the token as mentioned there

we do see below messages, but it seems to be a warning ?

registry-harbor-harbor-core-5475fdd75c-prgrj core 2023-07-14T05:26:13Z [WARNING] [/pkg/registry/auth/bearer/cache.go:99]: token issued time is empty, return expired to refresh token
registry-harbor-harbor-core-5475fdd75c-prgrj core 2023-07-14T05:26:13Z [WARNING] [/pkg/registry/auth/bearer/cache.go:99]: token issued time is empty, return expired to refresh token
registry-harbor-harbor-core-5475fdd75c-prgrj core 2023-07-14T05:29:36Z [WARNING] [/pkg/registry/auth/bearer/cache.go:99]: token issued time is empty, return expired to refresh token
registry-harbor-harbor-core-5475fdd75c-prgrj core 2023-07-14T05:29:38Z [WARNING] [/pkg/registry/auth/bearer/cache.go:99]: token issued time is empty, return expired to refresh token
registry-harbor-harbor-core-5475fdd75c-prgrj core 2023-07-14T05:29:38Z [WARNING] [/pkg/registry/auth/bearer/cache.go:99]: token issued time is empty, return expired to refresh token
registry-harbor-harbor-core-5475fdd75c-prgrj core 2023-07-14T05:29:42Z [WARNING] [/pkg/registry/auth/bearer/cache.go:99]: token issued time is empty, return expired to refresh token
registry-harbor-harbor-core-5475fdd75c-prgrj core 2023-07-14T05:29:43Z [WARNING] [/pkg/registry/auth/bearer/cache.go:99]: token issued time is empty, return expired to refresh token

[ugurmendi]

Hello I am still trying to solve the issue and the status is same here,If I deploy one replica it works like charm but If I increase it the problem start again ,
2023-07-17T06:22:16Z [ERROR] [/pkg/token/token.go:68]: parse token error, token is not valid yet
2023-07-17T06:22:16Z [WARNING] [/server/middleware/security/v2_token.go:53][requestID="e087ee6a4123bb84bdc1e066660e64ba"]: failed to decode bearer token: token is not valid yet
error code occured something like this

[shapirus]

I have this issue too. kops-managed k8s running on AWS EC2, must be good time-wise, yet probably miniscule differences affect this.
Maybe implementing loose token timestamp matching (e.g. accept anything within a +/- N sec window) could solve this?

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.