Allow creating a list of files shipped in a release

[pietroalbini]

This PR adds the BUILD_MANIFEST_SHIPPED_FILES_PATH environment variable to build-manifest, which writes a list of all the files referenced in the manifest to the path defined in the variable. The use for this is for promote-release to prune files unused files before publishing a release.

This PR does not implement any pruning, it just adds support for it to be implemented in the future on promote-release's side.

r? @Mark-Simulacrum

[pietroalbini]

To expand a bit on what the outcome of this effort is, this is a diff from the list of files shipped today and the ones that are not referenced in the manifest:

- build-manifest-nightly-x86_64-unknown-linux-gnu.tar.gz
- build-manifest-nightly-x86_64-unknown-linux-gnu.tar.xz
  cargo-nightly-x86_64-unknown-linux-gnu.tar.gz
  cargo-nightly-x86_64-unknown-linux-gnu.tar.xz
  channel-rust-nightly-date.txt
  channel-rust-nightly-git-commit-hash.txt
  channel-rust-nightly.toml
  clippy-nightly-x86_64-unknown-linux-gnu.tar.gz
  clippy-nightly-x86_64-unknown-linux-gnu.tar.xz
- cpu-dist-x86_64-linux.csv
- image-dist-x86_64-linux.txt
  llvm-tools-nightly-x86_64-unknown-linux-gnu.tar.gz
  llvm-tools-nightly-x86_64-unknown-linux-gnu.tar.xz
  miri-nightly-x86_64-unknown-linux-gnu.tar.gz
  miri-nightly-x86_64-unknown-linux-gnu.tar.xz
  rls-nightly-x86_64-unknown-linux-gnu.tar.gz
  rls-nightly-x86_64-unknown-linux-gnu.tar.xz
  rust-analysis-nightly-x86_64-unknown-linux-gnu.tar.gz
  rust-analysis-nightly-x86_64-unknown-linux-gnu.tar.xz
  rust-analyzer-nightly-x86_64-unknown-linux-gnu.tar.gz
  rust-analyzer-nightly-x86_64-unknown-linux-gnu.tar.xz
  rustc-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
  rustc-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
  rustc-docs-nightly-x86_64-unknown-linux-gnu.tar.gz
  rustc-docs-nightly-x86_64-unknown-linux-gnu.tar.xz
- rustc-nightly-src.tar.gz
- rustc-nightly-src.tar.xz
  rustc-nightly-x86_64-unknown-linux-gnu.tar.gz
  rustc-nightly-x86_64-unknown-linux-gnu.tar.xz
- rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
- rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
  rust-docs-nightly-x86_64-unknown-linux-gnu.tar.gz
  rust-docs-nightly-x86_64-unknown-linux-gnu.tar.xz
  rustfmt-nightly-x86_64-unknown-linux-gnu.tar.gz
  rustfmt-nightly-x86_64-unknown-linux-gnu.tar.xz
  rust-nightly-x86_64-unknown-linux-gnu.tar.gz
  rust-nightly-x86_64-unknown-linux-gnu.tar.xz
  rust-src-nightly.tar.gz
  rust-src-nightly.tar.xz
  rust-std-nightly-x86_64-unknown-linux-gnu.tar.gz
  rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz
- toolstates-linux.json

Of those, only rustc-nighly-src is a tarball we still want to ship (and we'd need to whitelist in the promote-release implementation), while all the other ones are either "internal" tarballs or extra files that don't matter in a release. If we were to implement this, another effect would be that nightly-only tarballs (like miri or rust-analyzer) would be removed from stable and beta releases while still being present on nightly.

[Mark-Simulacrum]

r=me

Is there a reason rustc-nightly-src is not included in the manifest already? In the long run I'd prefer that manifests were a 1:1 correspondence to the files we ship, i.e., nothing extra would be included -- beyond maybe internal things (e.g., rust-dev component which currently just contains llvm stuff).

[pietroalbini]

Is there a reason rustc-nightly-src is not included in the manifest already?

rustc-channel-src is not included in the manifest as it can't be installed through rustup. The tarball doesn't follow the structure rustup needs to be able to install it, but it's just a tarball of the (vendored) source code:

rust-src-1.47.0.tar.xz
├── components
├── install.sh
├── rust-installer-version
└── rust-src
    ├── lib
    │   └── rustlib
    │       └── src
    │           └── rust
    │               ├── Cargo.lock
    │               └── library/
    └── manifest.in

rustc-1.47.0-src.tar.xz
├── Cargo.lock
├── Cargo.toml
├── config.toml.example
├── configure
├── CONTRIBUTING.md
├── COPYRIGHT
├── git-commit-hash
├── library/
├── LICENSE-APACHE
├── LICENSE-MIT
├── README.md
├── RELEASES.md
├── src/
├── vendor/
├── version
└── x.py

[pietroalbini]

@bors r=Mark-Simulacrum

[bors]

📌 Commit 0ee1e91 has been approved by Mark-Simulacrum

[bors]

⌛ Testing commit 0ee1e91 with merge 9f6c670...

[bors]

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 9f6c670 to master...