Make "Assemble stage1 compiler" orders of magnitude faster #96803
Conversation
|
(rust-highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
|
@bors try I'm also going to kick off a try build and would appreciate you downloading and checking that all the files are there as expected -- in particular the rust-docs tarball is of concern to me, since IIRC we use symlinks to get rustdoc to generate files right for that and this will make it such that we're producing symlinks in the 'image' directory that's uplifted into the tarball I believe. |
|
⌛ Trying commit 52298d7dbb3328e2260b0e144fefa62aab9b3bb4 with merge be7cf4e1c0cda74138f4984a7ffe048b0af38e44... |
Mark-Simulacrum
added
S-waiting-on-author
This comment has been minimized.
This comment has been minimized.
|
💔 Test failed - checks-actions |
This comment has been minimized.
This comment has been minimized.
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
|
@bors try |
|
⌛ Trying commit 5f4b174 with merge 28bece3f394dc1aee54d00d6bb81b388ef6225ea... |
|
☀️ Try build successful - checks-actions |
|
@Mark-Simulacrum I downloaded https://ci-artifacts.rust-lang.org/rustc-builds/28bece3f394dc1aee54d00d6bb81b388ef6225ea/rust-docs-nightly-x86_64-unknown-linux-gnu.tar.xz and everything looks roughly right, I don't see any symbolic links in the archive and there are plenty of files in @rustbot ready |
rustbot
added
S-waiting-on-review
|
No, not really. I was thinking you'd diff against the parent commit file list. Anyway, cursory check seems good enough to me to go ahead, we can go further. @bors r+ |
|
📌 Commit 5f4b174 has been approved by |
bors
added
S-waiting-on-bors
|
ah, this is why symlinks didn't regress - we already dereference the link: Line 1416 in 5f4b174 |
|
☀️ Test successful - checks-actions |
|
Finished benchmarking commit (87fd70c): comparison url. Summary: This benchmark run did not return any relevant results. If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf. @rustbot label: -perf-regression |
…r=Mark-Simulacrum Revert "Make "Assemble stage1 compiler" orders of magnitude faster" Reverts rust-lang#96803. This caused `llvm-tools-nightly` to fail when installing with `rustup-toolchain-install-master` because of the presence of symlinks. I'm not sure how the symlinks got in there, but revert the PR for now while I figure it out. r? `@Mark-Simulacrum` cc `@RalfJung`
…crum
Make "Assemble stage1 compiler" orders of magnitude faster (take 2)
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
- Panic if we generate a symbolic link in a tarball
- Change install to use copy internally, like in my previous PR
- Change copy to dereference symbolic links, which avoids the previous regression in rust-lang#96803.
I also took the liberty of fixing `x dist llvm-tools` to work even if you don't call `x build` previously.
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
It turned out that
install()used full copies unconditionally. Change it to try using a hard-link before falling back to copying.