-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wire up PathMapper
in SpawnInputsExpander
#19718
Closed
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fmeum
force-pushed
the
6526-wire-up-spawninputsexpander
branch
from
October 4, 2023 09:43
8662630
to
333b562
Compare
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` in `SpawnInputsExpander`, which takes care of this for inputs to `Spawn`s executed in a sandbox or remotely. Constructs specific to Blaze, filesets and archived tree artifacts, are not covered by this change. Work towards bazelbuild#6526
fmeum
force-pushed
the
6526-wire-up-spawninputsexpander
branch
from
October 4, 2023 10:21
333b562
to
b7fdade
Compare
This was referenced Oct 4, 2023
github-actions
bot
added
awaiting-review
PR is awaiting review from an assigned reviewer
team-Configurability
platforms, toolchains, cquery, select(), config transitions
team-Performance
Issues for Performance teams
team-Remote-Exec
Issues and PRs for the Execution (Remote) team
labels
Oct 4, 2023
tjgq
reviewed
Oct 4, 2023
src/main/java/com/google/devtools/build/lib/analysis/actions/StrippingPathMapper.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/exec/SpawnInputExpander.java
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/exec/SpawnInputExpander.java
Show resolved
Hide resolved
tjgq
requested changes
Oct 5, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just one small nit on the tree artifact test, otherwise I think this is ready to be submitted.
src/test/java/com/google/devtools/build/lib/exec/SpawnInputExpanderTest.java
Outdated
Show resolved
Hide resolved
src/main/java/com/google/devtools/build/lib/exec/SpawnInputExpander.java
Show resolved
Hide resolved
tjgq
approved these changes
Oct 5, 2023
tjgq
added
awaiting-PR-merge
PR has been approved by a reviewer and is ready to be merge internally
and removed
awaiting-review
PR is awaiting review from an assigned reviewer
labels
Oct 5, 2023
github-actions
bot
removed
the
awaiting-PR-merge
PR has been approved by a reviewer and is ready to be merge internally
label
Oct 5, 2023
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 9, 2023
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` with sandbox outputs logic for sandboxed and worker sandboxed execution. An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721. Work towards #6526 Closes #19719. PiperOrigin-RevId: 571950903 Change-Id: I4a9d15da578da11f8677279b09e0864b8c33e9fc
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 16, 2023
`PathMapper`s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up the `PathMapper` used by a given `Spawn` in `RemoteExecutionService`, which ensures that paths of inputs and outputs are correctly mapped before being sent off to the remote executor and mapped back to the correct local paths when downloading the results. An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721. Work towards #6526 Closes #19721. PiperOrigin-RevId: 573806130 Change-Id: Ibbd4ff641eb301d78f5ec54813e65788d786fcea
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
team-Configurability
platforms, toolchains, cquery, select(), config transitions
team-Performance
Issues for Performance teams
team-Remote-Exec
Issues and PRs for the Execution (Remote) team
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PathMapper
s rewrite paths in command lines to make them more cache friendly, which requires executor support to stage files at the rewritten paths. This commit wires up thePathMapper
used by a givenSpawn
inSpawnInputsExpander
, which takes care of this for inputs toSpawn
s executed in a sandbox or remotely.Constructs specific to Blaze, filesets and archived tree artifacts, are not covered by this change.
An end-to-end test will be added in #18155, but requires #19718, #19719, and #19721.
Work towards #6526