Updating go.sum fails when a dependency is not resolvable with GOPROXY=direct

[simu]

What Renovate type, platform and version are you using?

Hosted app, on GitHub (github.com/projectsyn/boatswain).

Describe the bug

When dependencies of a Go project directly or indirectly reference a package which is not resolvable without the default GOPROXY settings (GOPROXY="https://proxy.golang.org,direct"), Renovate fails to update go.sum because go get -d ./... fails with a name resolution error.

Feel free to disregard this bug report, if this behavior is intended. I would appreciate a pointer to any workarounds if that's the case though.

Relevant debug logs

DEBUG: Executing command(branch="renovate/patch-aws-sdk-go")
{
  "command": "docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/projectsyn/boatswain\":\"/mnt/renovate/gh/projectsyn/boatswain\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/projectsyn/boatswain\" renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://github.com/\\\" && go get -d ./...\""
}
DEBUG: Failed to update go.sum(branch="renovate/patch-aws-sdk-go")
{
  "err": {
    "killed": false,
    "code": 1,
    "signal": null,
    "cmd": "docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/projectsyn/boatswain\":\"/mnt/renovate/gh/projectsyn/boatswain\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/projectsyn/boatswain\" renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://github.com/\\\" && go get -d ./...\"",
    "stdout": "",
    "stderr": "go: k8s.io/kubectl@v0.19.0 requires\n\tvbom.ml/util@v0.0.0-20160121211510-db5cfe13f5cc: unrecognized import path \"vbom.ml/util\": https fetch: Get \"https://vbom.ml/util?go-get=1\": dial tcp: lookup vbom.ml on 172.31.0.2:53: no such host\n",
    "message": "Command failed: docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/projectsyn/boatswain\":\"/mnt/renovate/gh/projectsyn/boatswain\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/projectsyn/boatswain\" renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://**redacted**@v0.19.0 requires\n\tvbom.ml/util@v0.0.0-20160121211510-db5cfe13f5cc: unrecognized import path \"vbom.ml/util\": https fetch: Get \"https://vbom.ml/util?go-get=1\": dial tcp: lookup vbom.ml on 172.31.0.2:53: no such host\n",
    "stack": "Error: Command failed: docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/projectsyn/boatswain\":\"/mnt/renovate/gh/projectsyn/boatswain\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/projectsyn/boatswain\" renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://**redacted**@v0.19.0 requires\n\tvbom.ml/util@v0.0.0-20160121211510-db5cfe13f5cc: unrecognized import path \"vbom.ml/util\": https fetch: Get \"https://vbom.ml/util?go-get=1\": dial tcp: lookup vbom.ml on 172.31.0.2:53: no such host\n\n    at ChildProcess.exithandler (child_process.js:303:12)\n    at ChildProcess.emit (events.js:311:20)\n    at ChildProcess.EventEmitter.emit (domain.js:482:12)\n    at maybeClose (internal/child_process.js:1021:16)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)"
  }
}

To Reproduce

Currently happens in github.com/projectsyn/boatswain (e.g. projectsyn/boatswain#41).

Unfortunately the PRs get automerged anyway, since the tests (which leave GOPROXY with its default value) pass.

Minimal example can also be found at https://github.com/simu/go-renovate-example. In the mean time, Renovate has also generated the same error as observed for projectsyn/boatswain: https://github.com/simu/go-renovate-example/pull/1

Additional context

I spent some time playing around with the docker command that can be seen failing in the Renovate debug logs. After reading up on go get, and seeing that GOPROXY is set to direct in the Docker image, I found that adding -e GOPROXY=https://proxy.golang.org,direct to the docker command resolves the issue.

The issue started appearing ~12 days ago (PRs projectsyn/boatswain#32 and projectsyn/boatswain#33).

[rarkins]

We should work out how to address this.

First of all if our go.sum updating fails then we should detect that and add a failing status check to the PR so that it can't be automerged.

I don't have my head around the proxy part though. Our "direct" approach means we bypass the proxy when we run? Do you think we should have a different default or just allow it to be configurable?

[simu]

I wasn't too familiar with the Go proxy either before investigating this issue. I found the documentation at https://proxy.golang.org/ and https://golang.org/cmd/go/#hdr-Module_downloading_and_verification as well as the output of go help goproxy quite helpful to get a basic understanding.

According to the documentation "proxy.golang.org is a module mirror which meets the spec provided at go help goproxy".

In the FAQ they state

Whenever possible, the mirror aims to cache content in order to avoid breaking builds for people that depend on your package, so this bad release may still be available in the mirror even if it is not available at the origin. The same situation applies if you delete your entire repository. We suggest creating a new version and encouraging people to use that one instead.

When you specify GOPROXY=direct you completely disable the module mirror and always try to fetch all modules from their repositories.

In my specific case however, it appears that the module vbom.ml/util cannot be fetched from https://vbom.ml as that domain does not resolve.

Since the module mirror has a cached copy of the module, the dependency can be fetched when GOPROXY is left at its default value of "https://proxy.golang.org,direct". With this configuration the mirror is tried first and only if the mirror returns 404 or 410, the direct module URL is tried.

[rarkins]

Thanks for looking into it. First of all, I assume it's a risky idea to be using a module that has disappeared but is still cached, right? Just want to check that I understand the situation.

I don't recall exactly why we set GOPROXY to direct, but I thought it was because private module resolution was failing. Is it possible that the first release didn't support the concept of falling back to direct?

The default value of "https://proxy.golang.org,direct" sounds like it should be backwards compatible with direct, but will we break anything if we switch to it?

[rarkins]

@moorara @acmcelwee do you have any recommendation here?

[simu]

Thanks for looking into it. First of all, I assume it's a risky idea to be using a module that has disappeared but is still cached, right? Just want to check that I understand the situation.

I agree that it's risky to use a module that cannot be resolved directly anymore, and would take steps to avoid this for direct dependencies. However, I don't have direct control over what dependencies the Kubernetes project (kubectl in this case) uses.

I don't recall exactly why we set GOPROXY to direct, but I thought it was because private module resolution was failing. Is it possible that the first release didn't support the concept of falling back to direct?

Apparently, private module resolution used to not work with GOPROXY not set to direct, but since go1.13 GOPROXY can be set to a comma-separated list of values, and there's a GOPRIVATE environment variable which can be set to a comma-separated list of globs for any private packages you might use. For example GOPRIVATE=*.internal.company.com.

The default value of "https://proxy.golang.org,direct" sounds like it should be backwards compatible with direct, but will we break anything if we switch to it?

If I understand https://golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules and https://golang.org/cmd/go/#hdr-Module_authentication_failures correctly, keeping GOSUMDB=off while having GOPROXY=https://proxy.golang.org,direct should be backwards compatible, as lookups of private modules on the proxy would result in a 404 and the next method (direct) would be tried in that case.

In any case I would think that exposing the whole Go proxy configuration would make sense, especially for self-hosted Renovate instances, as corporations may run their own internal Go proxies.

[rarkins]

If I have read our logic right, we pass the value of GOPROXY to our child process, and it passes the value to docker run if GOPROXY is defined. The renovate/go image we use when binarySource=docker default to GOPROXY=direct: https://github.com/renovatebot/docker-buildpack/blob/f876131c8c7c40a2b5a3083817b009fc2bde9298/src/golang/build/golang.sh#L34

For self-hosted users I think this means it will use their GOPROXY value already today.

In the hosted app, GOPROXY is not defined in the env so therefore is not passed to the image and retains the value direct.

It seems we could possibly resolve this issue by rebuilding our renovate/go containers with no GOPROXY variable defined and let it take the default value.

@viceice what do you think?

[viceice]

Sounds good

[viceice]

See PR renovatebot/docker-buildpack#52

[simu]

Should the linked change be rolled out by now? I got another failing PR (projectsyn/boatswain#45), which was originally created approximately 14 hours ago, but a retry just now still yields the same failure as originally observed:

From the renovate job #229113091 logs:

DEBUG: Fetching Docker image: docker.io/renovate/go:1.14.5(branch="renovate/patch-aws-sdk-go")
DEBUG: Finished fetching Docker image(branch="renovate/patch-aws-sdk-go")
DEBUG: Executing command(branch="renovate/patch-aws-sdk-go")
{
  "command": "docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/projectsyn/boatswain\":\"/mnt/renovate/gh/projectsyn/boatswain\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/projectsyn/boatswain\" docker.io/renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://github.com/\\\" && go get -d ./... && go mod tidy && go mod tidy\""
}
DEBUG: Failed to update go.sum(branch="renovate/patch-aws-sdk-go")
{
  "err": {
    "killed": false,
    "code": 1,
    "signal": null,
    "cmd": "docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/projectsyn/boatswain\":\"/mnt/renovate/gh/projectsyn/boatswain\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/projectsyn/boatswain\" docker.io/renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://github.com/\\\" && go get -d ./... && go mod tidy && go mod tidy\"",
    "stdout": "",
    "stderr": "go: k8s.io/kubectl@v0.19.0 requires\n\tvbom.ml/util@v0.0.0-20160121211510-db5cfe13f5cc: unrecognized import path \"vbom.ml/util\": https fetch: Get \"https://vbom.ml/util?go-get=1\": dial tcp: lookup vbom.ml on 172.31.0.2:53: no such host\n",
    "message": "Command failed: docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/projectsyn/boatswain\":\"/mnt/renovate/gh/projectsyn/boatswain\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/projectsyn/boatswain\" docker.io/renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://**redacted**@v0.19.0 requires\n\tvbom.ml/util@v0.0.0-20160121211510-db5cfe13f5cc: unrecognized import path \"vbom.ml/util\": https fetch: Get \"https://vbom.ml/util?go-get=1\": dial tcp: lookup vbom.ml on 172.31.0.2:53: no such host\n",
    "stack": "Error: Command failed: docker run --rm --name=renovate_go --label=renovate_child -v \"/mnt/renovate/gh/projectsyn/boatswain\":\"/mnt/renovate/gh/projectsyn/boatswain\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/tmp/renovate-cache/others/go\":\"/tmp/renovate-cache/others/go\" -e GOPATH -e CGO_ENABLED -w \"/mnt/renovate/gh/projectsyn/boatswain\" docker.io/renovate/go:1.14.5 bash -l -c \"git config --global url.\\\"https://**redacted**@github.com/\\\".insteadOf \\\"https://**redacted**@v0.19.0 requires\n\tvbom.ml/util@v0.0.0-20160121211510-db5cfe13f5cc: unrecognized import path \"vbom.ml/util\": https fetch: Get \"https://vbom.ml/util?go-get=1\": dial tcp: lookup vbom.ml on 172.31.0.2:53: no such host\n\n    at ChildProcess.exithandler (child_process.js:303:12)\n    at ChildProcess.emit (events.js:311:20)\n    at ChildProcess.EventEmitter.emit (domain.js:482:12)\n    at maybeClose (internal/child_process.js:1021:16)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)"
  }
}

[rarkins]

It seems we didn't succeed rebuilding all images before GitHub Actions ran out of space. Retrying manually now.

[rarkins]

We should now have them all rebuilt. Can you click the retry/rebase option and see if there's any change?

[simu]

Triggered a retry in projectsyn/boatswain#45. Will report back when the job has completed.

[simu]

Looks good now, thanks for the fast resolution 👍