random CI failures

[kulmann]

CI randomly fails in different pipelines with the following error message:

protoc \
		-I=third_party/ \
		-I=pkg/proto/v0/ \
		--go_out=pkg/proto/v0 settings.proto
	protoc-gen-go: program not found or is not executable
	Please specify a program using absolute path or make sure the program is available in your PATH system variable
	--go_out: protoc-gen-go: Plugin failed with status code 1.
	make: *** [Makefile:173: pkg/proto/v0/settings.pb.go] Error 1

The program not found or is not executable error appears randomly for:

• protoc-gen-go
• protoc-gen-micro
• protoc-gen-microweb

Example CI run: https://cloud.drone.io/owncloud/ocis-settings/106/3/4

Needs investigation.

[kulmann]

Same happens in ocis-accounts (e.g. https://cloud.drone.io/owncloud/ocis-accounts/387/2/5) and ocis-hello

[kulmann]

According to e.g. https://cloud.drone.io/owncloud/ocis-settings/277/1/7 protoc can't find protoc-gen-micro, protoc-gen-microweb etc.

Looking into our Makefile, those are installed into the $GOPATH:

ocis-settings/Makefile

Line 160 in 5249eec

$(GOPATH)/bin/protoc-gen-go:

Now looking at the docker image we're using:

• sets the $GOPATH as /srv/app, see https://github.com/dockhippie/golang/blob/b81c833b9a1eb5244e4c381368a40a9fd811a8ba/v1.13/Dockerfile.amd64#L18
• doesn't include $GOPATH/bin in $PATH, see https://github.com/dockhippie/golang/blob/b81c833b9a1eb5244e4c381368a40a9fd811a8ba/v1.13/Dockerfile.amd64#L19
  I'm wondering how this ever worked. Looking at the env vars, protoc is not able to find protoc-gen-micro by design.

Does anyone know why the $GOPATH is set to a directory that looks like it's supposed to serve a compiled binary and not have any other go binaries in it?

[kulmann]

On second thought, we check the generated proto code into our codebase anyway. Why do we need to run protoc at all? 🤔

[kulmann]

@C0rby see the comments here and this PR #54
Drone is still failing though, so changing from go get to go install didn't help.

[kulmann]

You can find the Dockerfile we're using here: https://github.com/dockhippie/golang/blob/master/v1.13/Dockerfile.amd64

[C0rby]

GOBIN is included to the PATH by /srv/app/bin https://github.com/dockhippie/golang/blob/master/v1.13/Dockerfile.amd64#L19. So that should not be the problem

[C0rby]

So I didn't really make any progress... it's strange but I think we should revert back from go install to go get because go install only compiles local projects and puts the binary into the $GOBIN directory.

My guess is that for some reason the task which does the go install or go get is not being triggered... but why?

[kulmann]

This issue has been resolved by explicitly including the protobuf step before build steps in .drone.star.
It still is odd and counts as a workaround, because we didn't find out why protoc is called in the build step in the first place. Forcing the protobuf step before build just makes sure that the required protoc plugins are installed properly. But it's not an explanation, why protoc runs during build...