remote_download_minimal: downloaded inputs shouldn't be deleted after the build

[brentleyjones]

Description of the problem / feature request:

When using --remote_download_minimal, downloaded inputs shouldn't be deleted after the build.

Feature requests: what underlying problem are you trying to solve with this feature?

Currently, when bazel needs to download inputs for a locally run action while using --remote_download_minimal, it deletes these inputs after the build:

bazel/src/main/java/com/google/devtools/build/lib/remote/RemoteModule.java

Lines 788 to 810 in 570f019

	/**
	* Delete any input files that have been fetched from the remote cache during the build. This is
	* so that Bazel's view of the output base is identical with the output base after a build i.e.
	* files that Bazel thinks exist only remotely actually do.
	*/
	private void deleteDownloadedInputs() throws IOException {
	if (actionInputFetcher == null) {
	return;
	}
	IOException deletionFailure = null;
	for (Path file : actionInputFetcher.downloadedFiles()) {
	try {
	file.delete();
	} catch (IOException e) {
	logger.atSevere().withCause(e).log(
	"Failed to delete remote output '%s' from the output base.", file);
	deletionFailure = e;
	}
	}
	if (deletionFailure != null) {
	throw deletionFailure;
	}
	}

This makes incremental compilation less efficient.

For example, if we set a bundling rule to run locally, bazel will download all of the various artifacts that need to be bundled together, and if we incrementally only change a single artifact in that bundle, bazel will have to re-download all of the artifacts, even though they didn't change since the last build. The fact that disk_cache doesn't work with remote builds ensure that this is a slow download as well.

What operating system are you running Bazel on?

macOS 10.15.7

What's the output of bazel info release?

release 4.0.0rc10

Have you found anything relevant by searching the web?

• BwtB deletes needed runfiles from output base #12635
• Separate downloads from action execution to provide more download options #12665

[coeuvre]

Fixed by f0c32de.

[brentleyjones]

Thank you!

[brentleyjones]

@coeuvre One "benefit" of this issue, that I've come to like since filing the bug, is that it allows large working sets to happen on space constrained machines (ones that the build would fail on before reaching the end, where one could call bazel clean). I wonder if we want this to be configurable in some way? And if so, could it be scoped (so we could apply it via a tag, and thus use --modify_execution_info)?

[coeuvre]

I don't think unconditionally delete downloaded intermediate outputs is a good way to manage the content of output base. bazel clean is a valid work around in this case since you will hit remote cache next time running bazel build and only download necessary files.

One way I could image is that we can collect the usages of these downloaded output files and delete those which are not used after a certain period of time.

[brentleyjones]

For my (new) specific use case, the build will fail if it doesn't delete the intermediate outputs during the build, since the local machine doesn't have enough space to have all of them at once. If it deletes as it goes the build succeeds. I'm not saying that my use case warrants a change, but this might count as a breaking change because of that?

[coeuvre]

deleteDownloadedInputs was called after a build command. I don't think we were able to delete intermediate outputs during the build.

[brentleyjones]

Hmm. I was seeing that the local inputs were deleted during the build. For example, we run tests locally, so the tests themselves are being downloaded, but then deleted right after each one is run (at least it looks that way).

[brentleyjones]

I'm running into an issue on HEAD (though it probably existed since this was fixed), where even though the inputs are no longer deleted, they aren't used on subsequent runs, causing them to be downloaded again. If I generate the inputs locally, so they aren't downloaded for the local action, they aren't downloaded on subsequent runs.

Of note for this, the inputs being downloaded are outputs (though not all the outputs) from previous actions in the graph.

[coeuvre]

Can you please create a new issue and maybe include a repro there?