Bazel at HEAD gets DEADLINE_EXCEEDED with remote strategy

[philwo]

I think this is a release blocker.

Reproduction steps:

# Start remote_worker:
$ rm -rf /usr/local/google/tmp/worker
$ mkdir /usr/local/google/tmp/worker
$ bazel build //src/tools/remote_worker
$ bazel-bin/src/tools/remote_worker/remote_worker --work_path=/usr/local/google/tmp

# Start build using remote strategy ("bazel" is Bazel built from HEAD)
$ bazel clean
$ bazel --host_jvm_args=-Dbazel.DigestFunction=SHA1 --blazerc=/dev/null build --spawn_strategy=remote --strategy=Javac=remote --remote_cache=localhost:8080 --remote_executor=localhost:8080 //src:bazel
[...]
WARNING: CppLink remote work failed (io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED).
ERROR: /usr/local/google/home/philwo/src/bazel/third_party/protobuf/3.2.0/BUILD:529:1: Linking of rule '//third_party/protobuf/3.2.0:protoc' failed: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED.
WARNING: Genrule remote work failed (io.grpc.StatusRuntimeException: CANCELLED).
WARNING: Javac remote work failed (io.grpc.StatusRuntimeException: CANCELLED).
WARNING: CppCompile remote work failed (io.grpc.StatusRuntimeException: CANCELLED).
WARNING: Javac remote work failed (io.grpc.StatusRuntimeException: CANCELLED).
WARNING: CppCompile remote work failed (io.grpc.StatusRuntimeException: CANCELLED).
WARNING: CppCompile remote work failed (io.grpc.StatusRuntimeException: CANCELLED).
WARNING: CppLink remote work failed (io.grpc.StatusRuntimeException: CANCELLED).
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 66.525s, Critical Path: 10.37s

[philwo]

FYI @ola-rozenfeld

[ola-rozenfeld]

The default minute of gRPC deadline should probably be increased, although it's the first time I see it not suffice for src:bazel. I don't think it's a release blocker, actually. But if you do, the best solution is increase the default timeout, imo.

[ulfjack]

Something is preventing the remote worker from responding in a timely manner, and we don't think that we should increase the default timeout without understanding what. Our hypothesis is that we're blocking the gRPC threads with action execution. If that's correct, we may need to change it to run actions in a separate thread pool.

[philwo]

If this is a server-side problem, I don't think it's a release blocker and we should remove the tag.
We don't really bundle or ship the remote worker with specific Bazel versions anyway.

If it's a client-side problem with our RemoteSpawnStrategy, it probably is a blocker for 0.5.2.

[ulfjack]

I haven't been able to reproduce so far.

[ulfjack]

I wonder if 716b527 may have fixed this accidentally?

I saw non-zero exit twice:

ERROR: /.../Projects/os-bazel/src/main/java/com/google/devtools/build/lib/BUILD:1205:1: Building Java resource jar failed: singlejar failed: error executing command external/bazel_tools/tools/jdk/singlejar/singlejar --normalize --dont_change_compression --exclude_build_data --output ... (remaining 19 argument(s) skipped): Exit -1

[ola-rozenfeld]

The non-zero exit is #3251, I believe. I still see it a lot.