Many tests failing on Buildkite nightly

[jin]

They all have the same error message:

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //configs/debian8_clang:debian8-clang-0.3.0-bazel_0.14.1-autoconfig_test
-----------------------------------------------------------------------------
++ main
+++ pwd
++ export PYTHON_RUNFILES=/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/7dff7d5699a88cff8b8587d4b4e0a89c/sandbox/linux-sandbox/524/execroot/bazel_toolchains/bazel-out/k8-fastbuild/bin/configs/debian8_clang/debian8-clang-0.3.0-bazel_0.14.1-autoconfig_test.runfiles/bazel_toolchains/../
++ PYTHON_RUNFILES=/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/7dff7d5699a88cff8b8587d4b4e0a89c/sandbox/linux-sandbox/524/execroot/bazel_toolchains/bazel-out/k8-fastbuild/bin/configs/debian8_clang/debian8-clang-0.3.0-bazel_0.14.1-autoconfig_test.runfiles/bazel_toolchains/../
++ trap cleanup_on_finish EXIT
++ mkdir ./project_src
++ tar -xf external/bazel_toolchains/rules/cc-sample-project-tar.tar -C ./project_src
tar: external/bazel_toolchains/rules/cc-sample-project-tar.tar: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
++ cleanup_on_finish
++ rm -drf ./project_src
++ rm outputs.tar
rm: cannot remove 'outputs.tar': No such file or directory
=== Deleting images  ===
Deleting all dangling images...

Please see https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/364#b650c23a-0a24-4f32-bf61-a070bbe9fc2f

ping @nlopezgi @xingao267

[xingao267]

Will take a look.

[xingao267]

I assume the nightly is built and tested using Bazel HEAD or some commit close Bazel HEAD. I tested it locally and they are find. These tests also pass in presubmit as well. Maybe something in Bazel between HEAD and 0.15.2 breaks this?

[jin]

Pulling in @philwo @buchgr, any ideas?

[philwo]

tar: external/bazel_toolchains/rules/cc-sample-project-tar.tar: Cannot open: No such file or directory

The "Bazel (with downstream projects)" uses a Bazel built from HEAD to test the downstream projects. So, considering that https://buildkite.com/bazel/bazel-toolchains/builds/668 is green but https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/367#3e3b5f1a-1c6e-4323-9d6b-bd0d2e6bc259 is red, I'd assume that something between Bazel 0.16.0 and Bazel HEAD broke.

Can you repro the failure when you build Bazel from HEAD and then use that Bazel to run the tests? If yes, I'd recommend a git bisect. I can help with that, but I have to fix another blocking bug for Nick first today and I'll be out of office tomorrow :|