Excessive re-building triggered by commit 27c983b (#2534)

[jrajahalme]

Title: Excessive re-building triggered by commit 27c983b (#2534)

Description:
commit 27c983b (PR #2534) seemingly works as intended as long as the bazel server is running. If the bazel server is restarted, however, some, if not most, dependencies are re-built, taking about 10 minutes CPU time.

This issue is easy to see with the bazel --batch option, as it does not leave the server running in the background.

Repro steps:

$ time bazel build //source/exe:envoy-static
... long build if nothing in cache ...
$ time bazel build //source/exe:envoy-static
... fast build, everything cached ...
$ bazel shutdown
$ time bazel build //source/exe:envoy-static
... long build, as if nothing is cached ...

Note that this behavior is not observed on previous commits, there bazel caching works also across bazel server restarts and in batch mode.

[jrajahalme]

Should add that none of the related environment variables are defined (CC CXX CFLAGS CXXFLAGS LD_LIBRARY_PATH). I have not tested with any of them defined.

[htuch]

I'll take an AI to look into this. Would be helpful if you could set them to "" and see if it still does this as a data point.

[jrajahalme]

Do you know if setting these env variables to "" is safe? Would that not override some defaults with nothing?

[jrajahalme]

Retested with export CFLAGS="" and export CXXFLAGS="", still the same problem: rebuilds after each bazel server restart. Makes incremental docker builds practically impossible.

[mattklein123]

@htuch FYI I just tried it locally and this is also broken for me with our external build script outside of bazel:

#!/bin/bash

set -e

DEPS=$(python <(cat envoy/bazel/target_recipes.bzl; \
  echo "print ' '.join(\"${THIRDPARTY_DEPS}/%s.dep\" % r for r in set(TARGET_RECIPES.values()))"))
echo "Building deps ${DEPS}"

envoy/ci/build_container/build_and_install_deps.sh ${DEPS}

ln -sf `pwd`/generated/thirdparty envoy/ci/prebuilt/thirdparty
ln -sf `pwd`/generated/thirdparty_build envoy/ci/prebuilt/thirdparty_build

Rebuilds for me every time. It used to detect no changes.