ScanAll not triggering for all images #18455
Comments
|
@dioguerra To your question:
The answer is |
|
To your question:
The daily scanner (scanAll) runs independently of
|
|
Yes. 1. was fixed by using stateless .cache (i dropped the volumeMount manually from the statefulset). For 2, i cannot do this as doing so and activating some sort of global scanning would start schedulling alot of jobs which would affect (sometimes halting) normal registry operation. Our experience shows that any overload of the jobService runners will affect registry image serving operations. #17607 Do you have any other recommendations? Note: I still need to validate this thread which seems to schedule scan jobs for the retrieved artifacts. Maybe there is something filtering happening here? harbor/src/controller/scan/base_controller.go Line 387 in f21b148 |
|
This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days. |
|
Still happening. The automatically daily scan job keeps scanning random number of images. See below: |
|
I also notice that scan_reports with no vulnerabilties have
shows 112 entries, althout most probably this also account recently pushed images that where pushed to projects with |
|
Continuing from #18455 (comment) The Scan is created with the SELECT count(id)
FROM public.artifact;
31094Inside the scan call we do some things:
At this point the
Observation. Shouldn't this be: supported := hasCapability(r, a)
if supported {
artifacts = append(artifacts, a) // If the artifact is not supported why do we add it to the list of artifacts?
scannable = true
return ar.ErrSkip // this artifact supported by the scanner, skip to walk its children
}
// What reason would require to add all other artifacts not supported by the scanner? Just to add a `not supported`?, we could add from hereSo, image indexes are not supported by trivy, if we filter the remainder artifacts we get: SELECT count(id)
FROM public.artifact
WHERE manifest_media_type IN ('application/vnd.oci.image.manifest.v1+json', 'application/vnd.docker.distribution.manifest.v2+json');
27445 |
|
I think this might also be causing The ScanAll report to be incorrectly aggregated? Does it make sense to exit? Shouldn't all reports be aggregated? for _, group := range groupReports {
if len(group) != 0 {
reports = append(reports, group...)
}
// else {
// // NOTE: If the artifact is OCI image, this happened when the artifact is not scanned,
// // but its children artifacts may scanned so return empty report
// return nil, nil
// }
} |
|
I'm seeing some errors when trivy tries to pull images from private repositories when triggered with It also seems that in version v2.7.x the job count is bigger (not sure this is due to the fact that trivy fails because s3 data is not there) Does this fail because of some sort of timeout? |
|
It seems that this problem is fixed in the version v2.7.x? I will need to actively scan the images to make sure: This is how i'm validating.
Here are some information on the scan jobs: SELECT count(*)
FROM public.artifactFrom this artifacts, 28639 should be scannable by the trivy scan: SELECT count(*)
FROM public.artifact AS a
LEFT JOIN public.scan_report AS sr
ON a.digest = sr.digest
WHERE
a.manifest_media_type IN ('application/vnd.oci.image.manifest.v1+json', 'application/vnd.docker.distribution.manifest.v2+json')Below its the last SCAN_ALL triggered (manually and schedule in the following order): SELECT id, vendor_type, vendor_id, status, status_message, trigger, extra_attrs, start_time, end_time, revision, update_time
FROM public.execution
WHERE vendor_type = 'SCAN_ALL'
ORDER BY id DESC
LIMIT 1000;Harbor v2.7 - automatic scan As we can observe, the total_count approaches the artifacts that trivy supports (~25k vs ~29k). I will try and test patch #18931 and #18943 to see the difference |
|
It seems that total_count seems much more consistent but even with the patches the reported scans show only a fraction of the total artifacts. In the table below, the last 2 reports where ran with the core v2.7 with the respective patches above NOTE: I disabled the GC but it seems that still some artifacts got dropped? (24745 -> 24682) Still this dosen't tell us nothing of what is failing. Checking the tasks for the currect execution id presents the same number of tasks and reports presented in the scan_all dashboard results (as we had confirmed before). SELECT *
FROM public.task
WHERE execution_id = 7148968Is this an issue submitting the task jobs from the SCAN_ALL artifact loop? Also, a last observation between the submit_count and the actual tasks assigned to the (latest) SCAN_ALL execution id they dont match:
You might say that this is because of the patches? But the same still happens with the previous original v2.7 version:
And for which i stopped as the total scheduled tasks where not increasing.
|
|
My logs show 22k occurences of this error harbor/src/controller/scan/base_controller.go Line 392 in f21b148 
which seems about right. Althought the full error is This is totally wrong. So what is the error exactly? |
Notes
|
|
I think the error comes from here. It points to a context timeout. [299] points to https://github.com/goharbor/harbor/blob/main/src/controller/scanner/base_controller.go#L301 |
|
This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days. |
|
This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue. |
Edit: Harbor Version v2.5.2 -- I know i know, update but please, read on...
So, I have a Daily ScanAll Job activated, which does not seem to be scanning all Objects. I did some investigation on the source code on my own, but seem to arrive to no conclusion.
First and foremost, what are the projects that the scanner triggers the scan jobs?
Automatically scan images on pushoption enabled?Personally I am expecting that 1. happens. Documentation is not explicit on this: https://goharbor.io/docs/2.7.0/administration/vulnerability-scanning/scan-all-artifacts/
A while back, our production image CVE scanner was not working aquasecurity/trivy#3894 (fixed now) and while new pushed images are being actively scanned, the daily scanner does not pick up the Images with a
Vulnerabilitieserror state:Even with
Automatically scan images on pushenabled. In this case, this is a proxy-cache repository.So, if we check the
Interrogation Service -> VulnerabilitiesI see ~ 2000 images scanned.Now, for the funzies part. I went through the src code starting with the scan_all.createOrUpdateScanAllSchedule call and arrived somewhere to:
harbor/src/controller/scan/base_controller.go
Line 371 in f21b148
harbor/src/controller/artifact/helper.go
Line 25 in f21b148
nil(as so are the options), so a new Query is instanciated with no filter parameters (only the pager) - Nothing much to seeharbor/src/controller/artifact/controller.go
Line 246 in f21b148
From what I managed to find out there dosent seem to be any filter for a specific artifact type, which I find quite suspitious?
I would expect the scan to use the manifest artifact and then the scanner itself(trivy) would pull all the dependent layers for scan. The vulnerabilities would then be aggregated and the vulnerability report would match the manifest only:
Here is an example of a query where one image is scanned and other is not:
NOTE: If the image first
artifactis an index, it seems that the scanner reports are tied to the underlaying manifests. So I infer that this should always be the case.So i have two cases. But even tho they are in the same repository, same image, the daily scan will not pick it up.
I would like to know what kind of query is made here, so I can reproduce how many images
would be scannedVSshould be scanned. It seems that some filtering is applyed but i didn't manage to pick it up.For consideration, This are my total artifact numbers:
AND
Note: number of artifacts obtained with commented parameter very similar
Can you help?
The text was updated successfully, but these errors were encountered: