x/vuln: govulncheck very slow in CI

[kian99]

govulncheck version

Go: go1.22.4
Scanner: govulncheck@v1.1.2
DB: https://vuln.go.dev
DB updated: 2024-07-03 16:27:09 +0000 UTC

Does this issue reproduce at the latest version of golang.org/x/vuln?

Yes, installed with go install golang.org/x/vuln/cmd/govulncheck@latest currently pointing at v1.1.2

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/runner/.cache/go-build'
GOENV='/home/runner/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/runner/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/runner/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/hostedtoolcache/go/1.22.4/x64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/hostedtoolcache/go/1.22.4/x64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/runner/work/<repo-name>/<repo-name>/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2116296393=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Ran govulncheck -show verbose ./... on a private repo via the golang/govulncheck-action@v1 Github action.
Also tested by SSHing into the machine and running the same command.

What did you see happen?

The output shows

Scanning your code and 1960 packages across 302 dependent modules for known vulnerabilities...

Fetching vulnerabilities from the database...

Checking the code against the vulnerabilities...

After 30 min there is still no further output. Locally the check finishes after ~1m30s but in CI on a Github runner the process seems to be taking extremely long.

What did you expect to see?

I expected to see the output after a few seconds/minutes. Is there any way to dig into what is being slow? Is the CI machine just too low-spec?

Some other info:

$ time go build ./...

real    8m7.259s
user    14m30.808s
sys     0m53.133s

Running govulncheck -mode binary takes <1s to run.

[gabyhelp]

Related Issues

• x/vuln: govulncheck ./... takes ~36mins #54940 (closed)
• x/vuln/cmd/govulncheck: consuming 90%+ of memory and runs without an end #67838 (closed)
• x/vuln/cmd/govulncheck: output a final result in json mode #62340 (closed)
• x/vuln: Missing output message when no vulnerabilities are found #61328 (closed)
• x/vuln: unexpected publishing latency #66872 (closed)
• x/vuln: feature request: govulncheck needs a (-f json) format option to output valid JSON for CI/CD tools to make decisions on the fly #60207 (closed)
• x/vuln/cmd/govulncheck: v1.1.0: Fatal error when used in a workspace #66863 (closed)
• x/vuln: use govulncheck in bazel #61494
• x/exp/vulndb/govulncheck: CVEs are not detected in GOPATH mode #51591 (closed)
• x/vuln: Cannot install govulncheck anymore since yesterday #57189 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[cagedmantis]

cc @golang/vulndb

[timothy-king]

Likely a dup of #54940 and #67838.

Is the CI machine just too low-spec?

Low memory is the most likely culprit. Do you have any information on how much govulncheck is consuming remotely?

Locally the check finishes after ~1m30s but in CI on a Github runner the process seems to be taking extremely long.

That is a bit surprising. How much memory is consumed locally?

Is there any way to dig into what is being slow?

@zpavlinovic any ideas?

[zpavlinovic]

This seems like a duplicate of #67838. We currently don't have a way of digging into the performance bottlenecks. This definitely seems to be a large repository (~2k dependent packages) so the memory consumption will spike to several GBs.

To validate this hypothesis, see if you can increase your CI memory limit. (My bet is that the current limit is 4GB, so I would bump it to 8GB)

[seankhliao]

Perhaps similar to #38714 govulncheck could get a flag/env for producing a runtime trace?

[kian99]

Thanks for the hints, I jumped into an SSH session on the CI machine and it has 8 GB of memory, running govulncheck ./... and checking htop I see that memory fills up all 8GB and then fills up 3GB of swap. I'll try a larger runner and see what happens. Locally I have a lot running so I'll try tomorrow with a more thorough analysis using valgrind and see how much memory is consumed.

Also, this repo is unfortunately private so I can't share a reproducer but I believe most of the package are coming from only a single import which is public so I'll look into that.

[zpavlinovic]

Perhaps similar to #38714 govulncheck could get a flag/env for producing a runtime trace?

Will add this to a to-look-into list.

[kian99]

Trying with Valgrind caused an SSL handshake timeout, so just running the check locally looks to use ~12 GB of memory. Our main dependency is https://github.com/juju/juju, running govulncheck ./... there results in similar memory usage (tested with their 3.5 branch).

Bumping up the CI machine to one with 16GB RAM resolved the problem. I'll close this issue, thank you for the help.