--bes_upload_mode=fully_async causes local BEP files to be written async too

[philwo]

Description of the problem / feature request:

When using --bes_upload_mode=fully_async, Bazel uploads BES asynchronously, so your bazel build command can exit, but the upload still continues in the background. This is nice for performance.

However, it seems to also cause local BEP files, e.g. --build_event_text_file or --build_event_json_file to be written asynchronously. This means that if you bazel build and save the BEP to a file, then immediately inspect the BEP, it might not be fully written yet.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

bazel clean
bazel build --nobuild --build_event_text_file=/tmp/bep.txt //...
grep targets_configured /tmp/bep.txt

The grep then often doesn't return anything, because the targets_configured thing is written at the end of the BEP file. If you add a sleep 3 before it, it usually works.

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

release 1.2.1

[brentleyjones]

What's the latest status on this? We parse the BEP as part of CI, but we want to use --bes_upload_mode=fully_async because it's a performance win.

[philwo]

That is an excellent question for @coeuvre! 😄

[coeuvre]

What is the expected behavior in this case? It seems to me that having local BEP files written completely before exiting Bazel contradicts to fully_async mode.

[brentleyjones]

The events that are generated that populate the BEP can be written to disk a lot faster than uploading them to BES. The desire is for --bes_upload_mode to only affect BES, not BEP (the name even says BES, not BEP).

For our use case, we upload the BEP in Buildkite, or do post-build logic based on what is in the BEP, but we are fine if BES continues to upload in the background.

[coeuvre]

I was thinking the case that, even we can write BEP to disk a lot faster, we still have to wait for local files be uploaded to remote server in order to get the converted URI (we can't get that ahead of time). So if we want to have local BEP files written completely before exiting Bazel, we have to wait for the file uploads.

[coeuvre]

We could probably make the URI in the local BEP files pointing to local files instead so there is no network traffic for generating local BEP files and can be blocked waiting after the build regardless --bes_upload_mode.

[brentleyjones]

We could probably make the URI in the local BEP files pointing to local files instead so there is no network traffic for generating local BEP files and can be blocked waiting after the build regardless --bes_upload_mode.

We would actually like that as an option 😍. BEP having local URI and BES having remote cache URI.

[brentleyjones]

Hi @coeuvre, just a ping on this after a couple months.

[coeuvre]

We could probably make the URI in the local BEP files pointing to local files instead so there is no network traffic for generating local BEP files and can be blocked waiting after the build regardless --bes_upload_mode.

IIUC, --build_event_json_file_path_conversion=false or --build_event_text_file_path_conversion=false does exactly this. Does that work in your case?

[brentleyjones]

That seems to work. As far as I can tell the BEP file is fully written to disk after the bazel command completes.

[coeuvre]

As far as I can tell the BEP file is fully written to disk after the bazel command completes.

Do you mean even with --build_event_json_file_path_conversion=false, the BEP json file is still not completed written before bazel command completes?

By looking at the code, I think it should be completed written before bazel command exits.

[brentleyjones]

No, I meant that with --build_event_json_file_path_conversion=false it looked like it is completely written to disk. That code seems to confirm it as well. I think this can be closed with this workaround.

[coeuvre]

@philwo Do we have action items on Bazel CI (or other places) to address this issue? Otherwise we can close this.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!