Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trivy fails on same image multiple hash scans #3894

Closed
dioguerra opened this issue Mar 23, 2023 · 4 comments
Closed

Trivy fails on same image multiple hash scans #3894

dioguerra opened this issue Mar 23, 2023 · 4 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@dioguerra
Copy link

dioguerra commented Mar 23, 2023 •
edited
Loading

Description

Trivy seems to fail when multiple run jobs inpacting the same layer run at the same time.
This happens on a kubernetes environment where the trivy workers are sharing the same PVC

adapter container version: v2.5.4 and also tried to upgrade to v2.7.1

What did you expect to happen?

Scans to be successfull

What happened instead?

If the scanning is updated on more than the number of trivy workers, some of them (if not most, fail)
image

Asking to scan 3 at a time = to the number of trivy workers, seems to work successfully (after reseting trivy and its backing pvc/pv volume data). Ex:
image

Output of run with -debug:

Example of a failed run:

2023-03-23T10:52:22Z [INFO] [/pkg/scan/job.go:385]: {
  "uuid": "6809b473-11fb-11eb-93ee-e6a720a2df22",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://harbor-trivy:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "access_credential": "[HIDDEN]",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.29.2",
  "create_time": "2020-10-19T13:08:24.486231Z",
  "update_time": "2022-02-23T10:12:32.887639Z"
}
2023-03-23T10:52:22Z [INFO] [/pkg/scan/job.go:385]: {
  "registry": {
    "url": "http://harbor-core:80",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 1009,
    "repository": "foo/bar",
    "tag": "",
    "digest": "sha256:b1868ce7da429df31e678fa1e8eba3c400cdf0e6e7a0721af281348ec2587a60",
    "mime_type": "application/vnd.docker.distribution.manifest.v2+json"
  }
}
2023-03-23T10:52:22Z [INFO] [/pkg/scan/job.go:167]: Report mime types: [application/vnd.security.vulnerability.report; version=1.1]
2023-03-23T10:52:22Z [INFO] [/pkg/scan/job.go:222]: Get report for mime type: application/vnd.security.vulnerability.report; version=1.1
2023-03-23T10:52:24Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T10:52:29Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T10:52:34Z [ERROR] [/pkg/scan/job.go:292]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 2: 2023-03-23T10:52:28.286Z	�[35mDEBUG�[0m	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2023-03-23T10:52:29.481Z	�[35mDEBUG�[0m	cache dir:  /home/scanner/.cache/trivy
2023-03-23T10:52:29.482Z	�[35mDEBUG�[0m	Skipping DB update...
2023-03-23T10:52:29.482Z	�[35mDEBUG�[0m	DB Schema: 2, UpdatedAt: 2023-03-23 06:07:45.782507817 +0000 UTC, NextUpdate: 2023-03-23 12:07:45.782507417 +0000 UTC, DownloadedAt: 2023-03-23 10:48:26.598494392 +0000 UTC
2023-03-23T10:52:29.492Z	�[34mINFO�[0m	Vulnerability scanning is enabled
2023-03-23T10:52:29.492Z	�[35mDEBUG�[0m	Vulnerability type:  [os library]
2023-03-23T10:52:29.492Z	�[34mINFO�[0m	Secret scanning is enabled
2023-03-23T10:52:29.492Z	�[34mINFO�[0m	If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-03-23T10:52:29.492Z	�[34mINFO�[0m	Please see also https://aquasecurity.github.io/trivy/v0.29.2/docs/secret/scanning/#recommendation for faster secret detection
2023-03-23T10:52:29.761Z	�[35mDEBUG�[0m	No secret config detected: trivy-secret.yaml
2023-03-23T10:52:29.925Z	�[35mDEBUG�[0m	Image ID: sha256:ebeadf901a6fc5237da1444f9040c739921d9eab9d2f0d2f3f618fc49bacb6d8
2023-03-23T10:52:29.926Z	�[35mDEBUG�[0m	Diff IDs: [sha256:99d3aaa12abfbe4f54565e1a2008ffe7c0275d3779d1d177833e7d3ecc020163 sha256:aee56cdbc43b681ad703727c1fe4c843ad3360a8c1856b6c16efcad87a6337fd sha256:5f2ce5cb1ea5d994771b7c2802322d475b19934882aff0dba72066037d4417b1 sha256:0a9f0908080c06ce8c46c7b80172e87f2d2f704580a2ecb90d89e6b2b989f186 sha256:95ff1f9eb45723fa6ab3fe6c5a236eccef0af30846ddad960981b833957f7b2f sha256:6cb9c232199ca3b4297f296a0ca801d1c187e4dcbb28905a70b35730050a74d2]
2023-03-23T10:52:29.926Z	�[35mDEBUG�[0m	Base Layers: [sha256:99d3aaa12abfbe4f54565e1a2008ffe7c0275d3779d1d177833e7d3ecc020163]
panic: invalid page type: 3: 10

goroutine 1 [running]:
go.etcd.io/bbolt.(*Cursor).search(0xc0019d9d68, {0xc0014a6eb0, 0x47, 0x50}, 0x0?)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/cursor.go:250 +0x28b
go.etcd.io/bbolt.(*Cursor).searchPage(0xc0019d9d68, {0xc0014a6eb0, 0x47, 0x50}, 0x0?)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/cursor.go:308 +0x166
go.etcd.io/bbolt.(*Cursor).search(0xc0019d9d68, {0xc0014a6eb0, 0x47, 0x50}, 0xc0019d9d20?)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/cursor.go:265 +0x1ca
go.etcd.io/bbolt.(*Cursor).seek(0xc0019d9d68, {0xc0014a6eb0?, 0x50?, 0xc0019d9d88?})
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/cursor.go:159 +0x48
go.etcd.io/bbolt.(*Bucket).Get(0xc00101ba80, {0xc0014a6eb0, 0x47, 0x50})
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/bucket.go:262 +0x85
github.com/aquasecurity/trivy/pkg/fanal/cache.FSCache.getBlob({0x7f5fdad09fff?, {_, _}}, _, {_, _})
	/home/runner/work/trivy/trivy/pkg/fanal/cache/fs.go:70 +0x7e
github.com/aquasecurity/trivy/pkg/fanal/cache.FSCache.MissingBlobs.func1(0xc001472e88?)
	/home/runner/work/trivy/trivy/pkg/fanal/cache/fs.go:164 +0x153
go.etcd.io/bbolt.(*DB).View(0xc000798a20?, 0xc0019da2b8)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/db.go:772 +0x82
github.com/aquasecurity/trivy/pkg/fanal/cache.FSCache.MissingBlobs({0xc001366240?, {0xc001362400?, 0xc000798720?}}, {0xc0014a68c0, 0x47}, {0xc000ed4900?, 0x47?, 0x47?})
	/home/runner/work/trivy/trivy/pkg/fanal/cache/fs.go:161 +0xfe
github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.Inspect({{0x3d789b8, 0xc001669fc0}, {0x7f5fda75ea38, 0xc000e3fd70}, {{{0x0, 0x0, 0x0}, {0xc00101a0c0, 0x3, 0x4}}}, ...}, ...)
	/home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:101 +0x506
github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact({{_, _}, {_, _}}, {_, _}, {{0xc0019059e0, 0x2, 0x2}, {0xc001905a20, ...}, ...})
	/home/runner/work/trivy/trivy/pkg/scanner/scan.go:110 +0x103
github.com/aquasecurity/trivy/pkg/commands/artifact.scan({_, _}, {{0xc00136a740, 0xc001de0540, {0x3d30d48, 0x6}, 0x0, 0x1, {0x7ffd9ce08ad2, 0x1a}}, ...}, ...)
	/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:513 +0x3fe
github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact(_, {_, _}, {{0xc00136a740, 0xc001de0540, {0x3d30d48, 0x6}, 0x0, 0x1, {0x7ffd9ce08ad2, ...}}, ...}, ...)
	/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:215 +0xc7
github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).ScanImage(_, {_, _}, {{0xc00136a740, 0xc001de0540, {0x3d30d48, 0x6}, 0x0, 0x1, {0x7ffd9ce08ad2, ...}}, ...})
	/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:174 +0x147
github.com/aquasecurity/trivy/pkg/commands/artifact.run({_, _}, {{0xc00136a740, 0xc001de0540, {0x3d30d48, 0x6}, 0x0, 0x1, {0x7ffd9ce08ad2, 0x1a}}, ...}, ...)
	/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:354 +0x86e
github.com/aquasecurity/trivy/pkg/commands/artifact.Run(0xc00136a740, {0x34c3410, 0x5})
	/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:329 +0x168
github.com/aquasecurity/trivy/pkg/commands/artifact.ImageRun(0xc0013365a0?)
	/home/runner/work/trivy/trivy/pkg/commands/artifact/image.go:69 +0x25
github.com/urfave/cli/v2.(*Command).Run(0xc0013365a0, 0xc0018ddb80)
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.8.1/command.go:169 +0x631
github.com/urfave/cli/v2.(*App).RunContext(0xc000b429c0, {0x3d637e8?, 0xc000144020}, {0xc000134000, 0x11, 0x12})
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.8.1/app.go:341 +0xbc8
github.com/urfave/cli/v2.(*App).Run(...)
	/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.8.1/app.go:247
main.main()
	/home/runner/work/trivy/trivy/cmd/trivy/main.go:16 +0x4f
: general response handler: unexpected status code: 500, expected: 200

Related error, but slightly different (case, multiple same image hash scan requests)

2023-03-23T11:02:57Z [INFO] [/pkg/scan/job.go:385]: {
  "uuid": "6809b473-11fb-11eb-93ee-e6a720a2df22",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://harbor-trivy:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "access_credential": "[HIDDEN]",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.29.2",
  "create_time": "2020-10-19T13:08:24.486231Z",
  "update_time": "2022-02-23T10:12:32.887639Z"
}
2023-03-23T11:02:57Z [INFO] [/pkg/scan/job.go:385]: {
  "registry": {
    "url": "http://harbor-core:80",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 1009,
    "repository": "foo/bar",
    "tag": "",
    "digest": "sha256:62758fceee7350c257dc635251de1a8992aa4672563791d99f51730fae1703eb",
    "mime_type": "application/vnd.docker.distribution.manifest.v2+json"
  }
}
2023-03-23T11:02:57Z [INFO] [/pkg/scan/job.go:167]: Report mime types: [application/vnd.security.vulnerability.report; version=1.1]
2023-03-23T11:02:57Z [INFO] [/pkg/scan/job.go:222]: Get report for mime type: application/vnd.security.vulnerability.report; version=1.1
2023-03-23T11:02:59Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T11:03:04Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T11:03:09Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T11:03:14Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T11:03:19Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T11:03:24Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T11:03:29Z [INFO] [/pkg/scan/job.go:243]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-03-23T11:03:34Z [ERROR] [/pkg/scan/job.go:292]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 2: 2023-03-23T11:03:17.854Z	�[35mDEBUG�[0m	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2023-03-23T11:03:28.836Z	�[35mDEBUG�[0m	cache dir:  /home/scanner/.cache/trivy
2023-03-23T11:03:28.836Z	�[35mDEBUG�[0m	Skipping DB update...
2023-03-23T11:03:28.837Z	�[35mDEBUG�[0m	DB Schema: 2, UpdatedAt: 2023-03-23 06:07:45.782507817 +0000 UTC, NextUpdate: 2023-03-23 12:07:45.782507417 +0000 UTC, DownloadedAt: 2023-03-23 10:48:26.598494392 +0000 UTC
2023-03-23T11:03:28.859Z	�[34mINFO�[0m	Vulnerability scanning is enabled
2023-03-23T11:03:28.859Z	�[35mDEBUG�[0m	Vulnerability type:  [os library]
2023-03-23T11:03:28.859Z	�[34mINFO�[0m	Secret scanning is enabled
2023-03-23T11:03:28.859Z	�[34mINFO�[0m	If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2023-03-23T11:03:28.859Z	�[34mINFO�[0m	Please see also https://aquasecurity.github.io/trivy/v0.29.2/docs/secret/scanning/#recommendation for faster secret detection
2023-03-23T11:03:33.230Z	�[35mDEBUG�[0m	No secret config detected: trivy-secret.yaml
2023-03-23T11:03:33.418Z	�[35mDEBUG�[0m	Image ID: sha256:72d7351cae7b4d4ee55a8cf1e7099bc3d596b6f2ba2d7f58b2b55cc88fa478d4
2023-03-23T11:03:33.418Z	�[35mDEBUG�[0m	Diff IDs: [sha256:99d3aaa12abfbe4f54565e1a2008ffe7c0275d3779d1d177833e7d3ecc020163 sha256:d53d13602ee5994564685da65b3f45e51688a69a6031956f01695ad2bbf4b0fc sha256:5f2ce5cb1ea5d994771b7c2802322d475b19934882aff0dba72066037d4417b1 sha256:0a9f0908080c06ce8c46c7b80172e87f2d2f704580a2ecb90d89e6b2b989f186 sha256:0e1722481987bfdb25fc273568e15ff639d7ea8bdf3e486460f4735c9dcbeb8f sha256:0d72985a479ca0aa8eb9a24d62405ec30071642c1e2df9e3f83ffca3e7d0f248]
2023-03-23T11:03:33.418Z	�[35mDEBUG�[0m	Base Layers: [sha256:99d3aaa12abfbe4f54565e1a2008ffe7c0275d3779d1d177833e7d3ecc020163]
2023-03-23T11:03:33.452Z	�[35mDEBUG�[0m	Missing image ID in cache: sha256:72d7351cae7b4d4ee55a8cf1e7099bc3d596b6f2ba2d7f58b2b55cc88fa478d4
2023-03-23T11:03:33.452Z	�[35mDEBUG�[0m	Missing diff ID in cache: sha256:0d72985a479ca0aa8eb9a24d62405ec30071642c1e2df9e3f83ffca3e7d0f248
2023-03-23T11:03:33.452Z	�[35mDEBUG�[0m	Missing diff ID in cache: sha256:0e1722481987bfdb25fc273568e15ff639d7ea8bdf3e486460f4735c9dcbeb8f
2023-03-23T11:03:33.452Z	�[35mDEBUG�[0m	Missing diff ID in cache: sha256:d53d13602ee5994564685da65b3f45e51688a69a6031956f01695ad2bbf4b0fc
2023-03-23T11:03:33.452Z	�[35mDEBUG�[0m	Missing diff ID in cache: sha256:0a9f0908080c06ce8c46c7b80172e87f2d2f704580a2ecb90d89e6b2b989f186
panic: page 11 already freed

goroutine 204 [running]:
go.etcd.io/bbolt.(*freelist).free(0xc000e7f100, 0x51, 0x7fa12efdb000)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/freelist.go:175 +0x2c8
go.etcd.io/bbolt.(*node).spill(0xc000931d50)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/node.go:359 +0x216
go.etcd.io/bbolt.(*node).spill(0xc000931ce0)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/node.go:346 +0xaa
go.etcd.io/bbolt.(*Bucket).spill(0xc00109ae00)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/bucket.go:570 +0x33f
go.etcd.io/bbolt.(*Bucket).spill(0xc0014c01d8)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/bucket.go:537 +0x107
go.etcd.io/bbolt.(*Tx).Commit(0xc0014c01c0)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/tx.go:160 +0xe7
go.etcd.io/bbolt.(*DB).Update(0x3400180?, 0xc000d8d730)
	/home/runner/go/pkg/mod/go.etcd.io/bbolt@v1.3.6/db.go:748 +0xe5
github.com/aquasecurity/trivy/pkg/fanal/cache.FSCache.PutBlob({0xc0006cb8c0?, {_, _}}, {_, _}, {0x2, {0xc0020f3680, 0x47}, {0xc0020f2870, 0x47}, ...})
	/home/runner/work/trivy/trivy/pkg/fanal/cache/fs.go:85 +0x1c5
github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspect.func1({0x3d63820, 0xc0022bb4a0}, {0xc0020f2d20, 0x47})
	/home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:177 +0x3a6
created by github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspect
	/home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:163 +0x532
: general response handler: unexpected status code: 500, expected: 200

Output of trivy -v:

trivy -v                                                                                                                                                                                                  
Version: 0.29.2
Vulnerability DB:
  Version: 2
  UpdatedAt: 2023-03-23 06:07:45.782507817 +0000 UTC
  NextUpdate: 2023-03-23 12:07:45.782507417 +0000 UTC
  DownloadedAt: 2023-03-23 10:48:26.598494392 +0000 UTC

Additional details (base image name, container registry info...):
@dioguerra dioguerra added the kind/bug Categorizes issue or PR as related to a bug. label Mar 23, 2023
@dioguerra
Copy link
Author

I just tried disabling the VolumeMount on trivy pod and scanning multiple images seems to work great.

  • 3 trivy scanner pods
  • Around 50 Scan jobs launched
  • 1 Job per trivy pod

@knqyf263
Copy link
Collaborator

I'm unfamiliar with Harbor, but I think you can select Redis as the cache backend. Sharing the filesystem cache between multiple Trivy instances is not supported.

@dioguerra
Copy link
Author

This is actually what I did: not sure it is working tho. What are the keys that are injected into redis?

replace in trivy statefulset

image:
    * from: goharbor/trivy-adapter-photon:v2.5.4
    * to: goharbor/trivy-adapter-photon:v2.7.1
    
        
enf:
- name: SCANNER_TRIVY_SECURITY_CHECKS
  value: "vuln"

args:
- --scanners vuln
- --cache-backend redis://harbor-redis.prod.svc.cluster.local:6379


Drop: (after terminationMessagePolicy)
        volumeMounts:
        - mountPath: /home/scanner/.cache
          name: data

@dioguerra
Copy link
Author

Ok. After testing, i can confirm that this is in fact because different trivy instances where using the same data and this was causing issues. I will fix this where appropriate.

This was referenced Mar 24, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@knqyf263 @dioguerra