Unable to access private gcr images even have right service account defined to scanjob

[saurabh21316]

Hello Team,

I would really appreciate if someone can help to resolve the private gcr docker images issue.
ERROR:
{"level":"error","ts":"2023-06-08T16:35:25Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-8644d5d6fc","container":"limit-processer","status.reason":"Error","status.message":"2023-06-08T16:35:21.135Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: unable to initialize a docker scanner: 4 errors occurred:\n\t* unable to inspect the image (us.gcr.io/abc/xyz:28ddd33d77904d113a5451c): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* containerd socket not found: /run/containerd/containerd.sock\n\t* GET https://us.gcr.io/v2/abc/xyz/manifests/28ddd33d77904d113a5451c: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication\n\n\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

Few things I tried when deploying via helm:
1.) Via apply GCP service account directly to trivy-operator instead of using default sa.
serviceAccountName: abcd123
serviceAccount: abcd123

2.) I also tried to define via helm:
---> privateRegistryScanSecretsNames: {"trivy-system": "trivy-scanner"}

3.) Via adding GCP SA annotation to trivy-operator SA Account.

None of them worked.

I can confirm the secrets works for gcr when i tested locally with docker run command as indicated on the below page.
https://aquasecurity.github.io/trivy/v0.25.4/docs/advanced/private-registries/gcr/

But if I directly run scan pod and add GOOGLE_APPLICATION_CREDENTIALS via env to container along with secret volume mount then it works but then I don't see an option to apply env/volume mount via helm to scan job (e.g configmap)

I would highly appreciate if someone can point me to right direction.

Thanks

[chen-keinan]

@saurabh21316 have you tried using global secrets (option 4 ) https://aquasecurity.github.io/trivy-operator/v0.14.1/tutorials/private-registries/

[saurabh21316]

I have tried method 1st,4th and 5th but still no luck

in 4th method, i used "us.gcr.io". :(

The error we see now

{"level":"error","ts":"2023-06-14T21:12:16Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-system/scan-vulnerabilityreport-788598479c","container":"dev-postgrest-crm-internal-ro","status.reason":"Error","status.message":"2023-06-14T21:12:12.792Z\t\u001b[31mFATAL\u001b[0m\tflag error: registry flag error: the length of usernames and passwords must match\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

[chen-keinan]

@saurabh21316 how you create the secret , it should be in form of :

kubectl create secret docker-registry artcred --docker-server=<registry> --docker-username=<user> --docker-password=<password> --docker-email=<mail>

Please also review this issue #1279

[saurabh21316]

Thanks for your response @chen-keinan !!
I am trying 4th method as explained in the above link but using gcp service account

e.g

1.) echo -n "{
"type": "service_account",
"project_id": "your-project-id",
"private_key_id": "your-private-key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\nyour-private-key\n-----END PRIVATE KEY-----\n",
"client_email": "your-service-account-email@your-project-id.iam.gserviceaccount.com",
"client_id": "your-client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/your-service-account-email%40your-project-id.iam.gserviceaccount.com"
}" | base64

2.) echo -n '{"auths":{"us.gcr.io":{"auth":"OUTPUT"}}}' | base64
3.) then adding secret in trivy-system

kind: Secret
type: kubernetes.io/dockerconfigjson
apiVersion: v1
metadata:
name: private-gcr-imagepull-secret
namespace: trivy-system
annotations:
container.developers.google.com/docker-credential-gcr: "true"
data:
.dockerconfigjson:

4.) then defining it under helm:
operator:
privateRegistryScanSecretsNames: {"trivy-system":"private-gcr-imagepull-secret"}

I could be doing wrong but I need a solution where I can use service account.

Thanks!

[chen-keinan]

@saurabh21316 it is not supported with gcp service account see issue #1095, currently only with with registry credential , user/password

[saurabh21316]

@chen-keinan thanks for the update. I tried the given GCR method as well on this link:
https://aquasecurity.github.io/trivy-operator/v0.14.1/docs/vulnerability-scanning/managed-registries/

apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
iam.gke.io/gcp-service-account: trivy-operator-gsa@<PROJECT_ID>.iam.gserviceaccount.com
name: trivy-operator
namespace: trivy-system

via adding annotation of gcp sa to default username (trivy-operator) created by helm and it didn't work either.

Does it even work?

Thanks

[chen-keinan]

Yes , we got the instruction from community how make it work in production.

Why not to use user/password secret as suggested?

[saurabh21316]

I don't think there is an option to use user/password with gcr. Could you please share the instruction on how can I make service account work? @chen-keinan

[chen-keinan]

@saurabh21316 please confirm that following :

• You are running operator on GKE.
• Create an IAM service account for your application (gcr)
• Ensure that your IAM service account has the roles you need for pulling images
• bind iam service account with trivy-operator service account
• Annotate the Kubernetes service account with the email address of the IAM service account.
• Update your scan-job spec to schedule the workloads on nodes that use Workload Identity and to use the annotated Kubernetes service account.

[saurabh21316]

@chen-keinan As I said above this is exactly i did and it's not working. The same service account works when running using docker command locally. so iam service account has right permissions to access the gcr.

docker run -it --rm -v /tmp:/tmp
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/service_account.json
aquasec/trivy image gcr.io/your_special_project/your_special_image:your_special_tag

[saurabh21316]

The only concern for me is the instruction given on the above page is
https://aquasecurity.github.io/trivy-operator/v0.14.1/docs/vulnerability-scanning/managed-registries/

For Config Connector, apply the IAMServiceAccount object for your selected service account.
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
name: trivy-operator-gsa
spec:
displayName: trivy-operator-google-service-account

gcloud iam service-accounts add-iam-policy-binding trivy-operator-gsa@<GSA_PROJECT>.iam.gserviceaccount.com
--role roles/iam.workloadIdentityUser
--member "serviceAccount:<PROJECT_ID>.svc.id.goog[trivy-system/trivy-operator]"

I don't think this crd exist anymore.

[saurabh21316]

@chen-keinan please let me know if you get a chance to look at this.

[petskratt]

FYI I got latest Helm chart (--version 0.14.1) working with GCP gcr.io images using service-accounts after some hairpulling - main problem was container registry was under another project (e.g not the same one where trivy-operator was running).

Main steps according to doc eg. add serviceaccount, add policy-bindings for roles/iam.workloadIdentityUser + roles/storage.objectViewer (doc mentions "roles you need"), add annotation, set trivyOperator.scanJobAutomountServiceAccountToken to true

My custom-values.yaml:

serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account: <user-id>@<project-id>.iam.gserviceaccount.com

trivyOperator:
  scanJobAutomountServiceAccountToken: true

and then granting permissions on another project:

gcloud projects add-iam-policy-binding <another-project-id> \
--member serviceAccount:<user-id>@<project-id>.iam.gserviceaccount.com \
--role roles/storage.objectViewer

[chen-keinan]

FYI I got latest Helm chart (--version 0.14.1) working with GCP gcr.io images using service-accounts after some hairpulling - main problem was container registry was under another project (e.g not the same one where trivy-operator was running).

Main steps according to doc eg. add serviceaccount, add policy-bindings for roles/iam.workloadIdentityUser + roles/storage.objectViewer (doc mentions "roles you need"), add annotation, set trivyOperator.scanJobAutomountServiceAccountToken to true

@petskratt Thanks or sharing , do you mind reviewing our docs for gcp and let me know if anything need to be amended.

[saurabh21316]

yeah, we have it running in different project as well(e.g. abc-project). did you get it working? @chen-keinan

The service account I created have following roles under (abc-project) :
Artifact Registry Reader
Storage Object Viewer
Workload Identity User

So service account has right permissions as required and it looks like the procedure define here is broken. @petskratt would really appreciate if you can help on it as we have been stuck on it from last couple of weeks.

[chen-keinan]

@saurabh21316 the docs for gcp managed private registries has been contributed by community user who make it work same as @petskratt

[saurabh21316]

@chen-keinan I agree but as I said earlier, It seems like these crds don't exist anymore

resource mapping not found for name: "trivy-operator" namespace: "trivy-system" from "gke-service-account-connector.yaml": no matches for kind "IAMServiceAccount" in version "iam.cnrm.cloud.google.com/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "trivy-operator" namespace: "trivy-system" from "gke-service-account-connector.yaml": no matches for kind "IAMPolicy" in version "iam.cnrm.cloud.google.com/v1beta1"
ensure CRDs are installed first

here is the file:

---

apiVersion: v1
kind: ServiceAccount
metadata:
name: trivy-operator
namespace: trivy-system
annotations:
iam.gke.io/gcp-service-account: trivy-abc@abc-project.iam.gserviceaccount.com

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
name: trivy-operator
namespace: trivy-system
spec:
displayName: trivy-operator

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicy
metadata:
name: trivy-operator
namespace: trivy-system
spec:
resourceRef:
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
name: trivy-operator
bindings:
- role: roles/iam.workloadIdentityUser
members:
- serviceAccount:abc-project.svc.id.goog[trivy-system/trivy-operator]

[btwseeu78]

This idea of using it workload identity with trivy operator is security concern for a security tool, In GCP diff projects hosts diff repos,even individual repo has granual level permission.

The way it's going if you are running 50 application in a cluster, and if they belong to 50 diff repos then the access should be done for all these repo, giving this defies the basic principle of least privilege.
We would love to know what you guys think maybe,for adding service account support, according to that we will decide to use or drop trivy entirely.

[chen-keinan]

@btwseeu78 I'm not objecting for adding support for workload identity as describe above.
I might need help with this, happy to work it with you.

[btwseeu78]

@chen-keinan workload identity is nice way to access repository,but it was meant to be secure but for platform teams who hosts multiple project its burden because we dont control how individual project acesss their repo, as trivy works as of now that it uses the credentials from workload its scanning, this works for me but only if supports all the methods, since serviceaccount is one of the go to method for pulling from google artifact registry its generally used by many and easier, workload identity is not bad but for platform team we can not control how projects want to authenticate.

so i am wondering why the last issue was closed on this same topic, are we saying that serviceaccount in json will not be supported, thats diff problem since we require diff solution or finding way to fit it in. so the assurance tat we should wait or not should be at least available.

[chen-keinan]

@btwseeu78 are you referring to this issue #1095 , it was closed by github action stale bot probably missed it, now it is open. I'll take a look at it next week and see how to add support for it.

[saurabh21316]

@petskratt if you got it working when running trivy operator let's say in Project-A and then can access gcr image in Project-B then I must be doing something wrong. Going with the reply above as you said.
Let me explain you thing that I did for testing purpose
e.g.

Scenario 1:
a.) trivy operator running via helm in Project-A
b.) created service account(xyz) in Project-B with 3 roles as required:
Artifact Registry Reader
Storage Object Viewer
Workload Identity User
c.) applied the following setting in helm as you mentioned.
serviceAccount:
annotations:
iam.gke.io/gcp-service-account: @.iam.gserviceaccount.com

trivyOperator:
  scanJobAutomountServiceAccountToken: true

---

Scenario 2:
a.) trivy operator running via helm in Project-A
b.) created service account(abc) in Project-A with 3 roles as required:
Artifact Registry Reader
Storage Object Viewer
Workload Identity User

  --> Also added the service account(abc) in project-B and assigned the same roles in Project-B.

c.) applied the following setting in helm as you mentioned.
serviceAccount:
annotations:
iam.gke.io/gcp-service-account: @.iam.gserviceaccount.com

trivyOperator:
  scanJobAutomountServiceAccountToken: true

---

So I tried both method and none of it worked. Let me know if there is something I am doing wrong or missing the instruction.

[saurabh21316]

Ok I think I got it working guys.. The only piece was missing the below command since it was little confusing but eventually I see it's able to scan private gcr. will update it once it finishes scanning or I get into any error.

gcloud iam service-accounts add-iam-policy-binding trivy-operator-gsa@<GSA_PROJECT>.iam.gserviceaccount.com
--role roles/iam.workloadIdentityUser
--member "serviceAccount:<PROJECT_ID>.svc.id.goog[trivy-system/trivy-operator]"

Thanks for your support and continue response.

@petskratt @chen-keinan @btwseeu78

[chen-keinan]

@saurabh21316 thanks for the update , I would like to ask you the same, if all works , could you confirm if our gcr manages privare registry docs requires an update

[saurabh21316]

yes @chen-keinan imo, it's quite simple to get it working if instructions are updated because there is no need of connector and the CRD doesn't exist anymore.

[saurabh21316]

Happy to update the instructions if you like and we can go through it if require.

[saurabh21316]

To run the Trivy Operator in Project A while using Private Google Container Registry (GCR) in Project B, you need to follow these steps:

1.) Create an IAM service account in Project B with the necessary roles:

• Workload Identity User
• Artifact Registry Reader
• Storage Object Viewer

For example, it creates a service account named test-scanner@Project-B.iam.gserviceaccount.com with these roles assigned.

2.) Run the following command in the terminal, ensuring you are in Project B
(you can check using command via terminal):
--> gcloud config get project

gcloud iam service-accounts add-iam-policy-binding test-scanner@Project-B.iam.gserviceaccount.com
--role roles/iam.workloadIdentityUser
--member "serviceAccount:Project-A.svc.id.goog[trivy-system/trivy-operator]"

This command adds an IAM policy binding to the service account test-scanner@Project-B.iam.gserviceaccount.com with the role roles/iam.workloadIdentityUser and associates it with the service account trivy-system/trivy-operator in Project A and it's quite important to run this command otherwise it will not work.

3.) Update the custom-value.yaml file for the Trivy Operator before deploying it using Helm.

trivyOperator:
scanJobAutomountServiceAccountToken: true

serviceAccount:
create: true
annotations:
iam.gke.io/gcp-service-account: test-scanner@Project-B.iam.gserviceaccount.com

Ensure you make these modifications to the custom-value.yaml file specific to the Trivy Operator before deploying it via Helm.

By completing these steps, you will have created the necessary IAM service account, granted it the required roles, and configured the Trivy Operator with the appropriate service account annotation and settings for authentication and authorization between Project A and Project B.

[saurabh21316]

@chen-keinan i have just posted the fix that i did at my end however I am not sure if it requires when running in same project. please have a look.

[btwseeu78]

@btwseeu78 are you referring to this issue #1095 , it was closed by github action stale bot probably missed it, now it is open. I'll take a look at it next week and see how to add support for it.

i would try to code it myslef, im am new to go i will try to fix the issue.if not i would request someone to pick it up.

[btwseeu78]

To run the Trivy Operator in Project A while using Private Google Container Registry (GCR) in Project B, you need to follow these steps:

1.) Create an IAM service account in Project B with the necessary roles:

* Workload Identity User

* Artifact Registry Reader

* Storage Object Viewer

For example, it creates a service account named test-scanner@Project-B.iam.gserviceaccount.com with these roles assigned.

2.) Run the following command in the terminal, ensuring you are in Project B (you can check using command via terminal): --> gcloud config get project

gcloud iam service-accounts add-iam-policy-binding test-scanner@Project-B.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:Project-A.svc.id.goog[trivy-system/trivy-operator]"

This command adds an IAM policy binding to the service account test-scanner@Project-B.iam.gserviceaccount.com with the role roles/iam.workloadIdentityUser and associates it with the service account trivy-system/trivy-operator in Project A and it's quite important to run this command otherwise it will not work.

3.) Update the custom-value.yaml file for the Trivy Operator before deploying it using Helm.

trivyOperator: scanJobAutomountServiceAccountToken: true

serviceAccount: create: true annotations: iam.gke.io/gcp-service-account: test-scanner@Project-B.iam.gserviceaccount.com

Ensure you make these modifications to the custom-value.yaml file specific to the Trivy Operator before deploying it via Helm.

By completing these steps, you will have created the necessary IAM service account, granted it the required roles, and configured the Trivy Operator with the appropriate service account annotation and settings for authentication and authorization between Project A and Project B.

@chen-keinan @saurabh21316 just for the note to add, Workload identity is not a default enabled feature in GKE, This supposed be created preferred in times of node creation, else updating it is tricky for existing nodes.
there some known issues preferred it should be checked and added as requirement.

[cdenneen]

I have the imagePullSecrets set on the ServiceAccount but still getting issues.

imagePullSecrets:
- name: creds-docker

[saurabh21316]

It doesn't work @cdenneen therefore use the above method to access private gcr.

[cdenneen]

I opened #1364 the issue is I need this to work globally vs just a particular namespace right? or should it just be:

operator:
    privateRegistryScanSecretsNames: {"trivy-system":"creds-docker"}

all examples are for individual namespaces but if the operator spins all the images up in the trivy-system namespace then this would work I'm guessing and not sure why we'd need something like:

operator:
    privateRegistryScanSecretsNames: {"app":"dockerconfigjson-github-com"}

[btwseeu78]

Sry I haven't able to put my efforts on this, so the problem here is tricky uses special argument when you use google application credentials, it's an argument where you provide file path to the command GOOGLE_APPLICATION_CREDENTIALS=/path/to/credential.json. Now the way operator works currently does not support this argument it tries to convert it to docker auth which is not right for json creeds, we need a new volume mount when trivia docker auth finds a username == -json-key. And we also need to Mindy arts list that operator runs making username blank that should solve issues with private registry GCR and AR,popping the jobs or using workload identify can solve but only will work when your entire organisation pull from same private repository but for organisations who uses granular perm it's an headache as you will end up with hundreds of pull secrets append and managing @chen-keinan what your though I found the problem but not the time to fix it, maybe you can put your thought on this

[btwseeu78]

string{
"trivy",
},
Args: []string{
"--cache-dir",
"/var/trivyoperator/trivy-db",
"image",
"--download-db-only",
"--db-repository",
dbRepository,
},

[chen-keinan]

@btwseeu78 thanks for update , I'll take e a look at it

[btwseeu78]

I am not the best or the brightest in terms of go but i tested my changes with private registry with json creds works just fine as i thought it should be.

anybody wants to use this can test using : linuxarpan/trivy-test:v2 , @chen-keinan im not the best guy in terms go and overall dev i would let you handle the rest part.

[chen-keinan]

Thanks , Ill take a look

[github-actions]

This issue is stale because it has been labeled with inactivity.