fix: use git-commit-info for version information #100557

dawnofmidnight · 2022-08-15T02:34:45Z

This PR changes the current git-commit-hash file that ./x.py dist puts in the rustc-{version}-src.tar.{x,g}z to contain the hash, the short hash, and the commit date from which the tarball was created, assuming git was available when it was. It uses this for reading the version so that rustc has all the appropriate metadata.

Testing

Testing this is kind of a pain. I did it with something like

./x.py dist # ensure that `ignore-git` is `false` in config.toml
cp ./build/dist/rustc-1.65.0-dev-src.tar.gz ../rustc-1.65.0-dev-src.tar.gz
cd .. && tar -xzf rustc-1.65.0-dev-src && cd rustc-1.65.0-dev-src
./x.py build

Then, the output of rustc -vV with the stage1 compiler should have the commit-hash and commit-date fields filled, rather than be unknown. To be completely sure, you can use rustc --sysroot with the stdlib that the original ./x.py dist made, which will require that the metadata matches.

rust-highfive · 2022-08-15T02:34:48Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Mark-Simulacrum (or someone else) soon.

Please see the contribution instructions for more information.

dawnofmidnight · 2022-08-15T02:54:21Z

r? @jyn514

Mark-Simulacrum · 2022-08-15T02:57:10Z

Hm, so my initial thought here is that we should probably still embed some indication that the rustc isn't actually being built from that git commit, since otherwise it'll be pretty confusing when a distro patch accidentally or intentionally changes behavior (even just in terms of what can link together successfully).

Can you say more about what motivates this? It looks like based on the fixed issues largely to be a case of using srlo std binaries with a distro rustc, but that feels like a pretty odd use case to me (why not just use rustup entirely at that point), and I'm wary that adding rudimentary support is enough to get people into trouble later. At the very least I'd like to know more about the desire and think about it.

dawnofmidnight · 2022-08-15T03:11:23Z

Hello. I do agree that using an srlo binary just for std would indeed be strange and uncommon (probably why that first issue has been open for such a long time). I do think, however, that it's better to have more complete version information for binaries compiled from the tarballs. I suppose that that could be harmful if the tarball was changed and someone assumed it was the same as the actual commit, though. Perhaps jyn514 has some other thoughts on that as well.

Hm, so my initial thought here is that we should probably still embed some indication that the rustc isn't actually being built from that git commit

How do you think this could look, if this was done? I do agree that that's probably a good thing to have in there, but I'm not sure how.

On another note, I'll try to fix up those formatting issues tomorrow morning, I completely forgot to run that.

jyn514 · 2022-08-15T09:40:09Z

Hm, so my initial thought here is that we should probably still embed some indication that the rustc isn't actually being built from that git commit

How do you think this could look, if this was done? I do agree that that's probably a good thing to have in there, but I'm not sure how.

We could put this in --version --verbose maybe. I think in practice it won't be a giant deal; if distros are applying in patches, they should already be using rust.description.

Can you say more about what motivates this? It looks like based on the fixed issues largely to be a case of using srlo std binaries with a distro rustc, but that feels like a pretty odd use case to me (why not just use rustup entirely at that point), and I'm wary that adding rudimentary support is enough to get people into trouble later.

"We shouldn't arbitrarily limit what people can compile". If the people build from source without patches, that's effectively the same compiler we distribute on static.rlo and there's no reason to prevent people from using it with the precompiled artifacts.

That said, I agree it's pretty strange to want to do this - would love to hear from @japaric or @Zapeth why they wanted this feature originally. (I still think it's very useful to embed the commit metadata somehow even if we don't guarantee metadata compatibility, though.)

Mark-Simulacrum · 2022-08-15T10:41:43Z

I think a reasonable compromise would be to set rust.description to include "tarball", by default, with a comment recommending how to avoid git-commit-hash being populated if building with patches. If we have that there I think that's enough for most of my concerns to be addressed.

Even building from source without patches is a little on the questionable side (e.g., there's going to be divergence in performance due to lack of PGO and other opts we run). I think building std from source is something I'm less worried about (ultimately required for many use cases and we intend to officially support that in the future), but re-compiling rustc and then loading std from srlo seems odd.

jyn514 · 2022-08-15T15:15:16Z

I think a reasonable compromise would be to set rust.description to include "tarball", by default, with a comment recommending how to avoid git-commit-hash being populated if building with patches. If we have that there I think that's enough for most of my concerns to be addressed.

👍 seems good to me, we still let people do this but they have to opt into it intentionally.

@dawnofmidnight could you make that change please? I would expect the code for it to be somewhere near the changes you've already made, but I haven't looked at it.

Zapeth · 2022-08-15T15:15:40Z

I'm not sure what other information I can provide aside from what was already written in the linked issues, but from what I can remember my use case was that I wanted to cross-compile my rust code with a distribution-provided toolchain (ie built from tarball).

I agree that adding an indication in the version information about rustc possibly not corresponding to upstream source would be useful. Maybe this could be added as an optional postfix to the git-commit-hash, using an environment variable when building the toolchain? (assuming this doesn't conflict again with targeting srlo binaries).

jyn514 · 2022-08-15T15:20:36Z

Maybe this could be added as an optional postfix to the git-commit-hash, using an environment variable when building the toolchain?

This is what I meant with rust.version above, yeah. We already support it today :)

assuming this doesn't conflict again with targeting srlo binaries

No, this is incompatible with patches. If you have a different compiler, it cannot reuse the same metadata, patches can change metadata handling in arbitrary ways. You can either patch the compiler, or you can reuse pre-compiled libraries, but you can't do both.

dawnofmidnight · 2022-08-15T17:34:12Z

@jyn514 I'm a little confused, what exactly do you mean by rust.{version,description}? Do you mean setting something like this: https://github.com/dawnofmidnight/rust/blob/tarball-commit-info/src/bootstrap/config.rs#L181 (it seems this is controlled by a field in config.toml, if I understand correctly)? If so, would that show up in rustc -vV? Sorry, I'm not too familiar with boostrap and how it interacts with rustc.

jyn514 · 2022-08-15T23:29:41Z

@dawnofmidnight yes, those fields in config.toml are exactly what I mean :) you can read their documentation in config.toml.example. Both show up in --version iirc, which means they also affect whether metadata can be reused between compilers.

dawnofmidnight · 2022-08-15T23:52:38Z

Both show up in --version iirc, which means they also affect whether metadata can be reused between compilers.

Got it, thank you. I guess I'll append something like "Built from a source tarball." to the description.

bors · 2022-08-21T02:23:06Z

☔ The latest upstream changes (presumably #99967) made this pull request unmergeable. Please resolve the merge conflicts.

dawnofmidnight · 2022-08-23T17:11:17Z

I've got a local fix that makes rustc -V look like

rustc 1.65.0-dev (a785176 2022-08-22) (built from a source tarball)

Does that look okay? I'll get it pushed once I figure out these git issues.

jyn514 · 2022-08-24T01:49:31Z

@dawnofmidnight yes, that looks fine :) thanks!

dawnofmidnight · 2022-08-24T17:12:43Z

Okay, I think I've gotten everything figured out wrt git rebasing, but please let me know if there's something I messed up.

dawnofmidnight · 2022-08-24T17:19:45Z

...it appears not. That error doesn't seem to exist locally, strangely. I'll try again and make sure everything is fine.

rustbot · 2022-08-24T17:40:31Z

Some changes occurred in src/tools/cargo

cc @ehuss

dawnofmidnight · 2022-08-24T17:45:43Z

That wasn't supposed to happen. Maybe I screwed something up while rebasing.

I'm not sure how to fix this, so I'm just going to leave it alone for a bit so I don't mess anything else up.

jyn514

You should be able to undo the cargo changes by running git checkout HEAD~2 src/tools/cargo && git commit --amend.

This looks mostly right to me, but I think we're no longer solving the original issue; I left a review comment.

src/bootstrap/channel.rs

src/bootstrap/config.rs

src/bootstrap/dist.rs

src/bootstrap/lib.rs

jyn514

This looks great, thanks :) I assigned this to @Mark-Simulacrum since he had opinions on the approach earlier, but r=me if Mark doesn't have comments.

config.toml.example

src/bootstrap/tarball.rs

Mark-Simulacrum · 2022-09-17T15:17:09Z

@rustbot author

dawnofmidnight · 2022-09-29T04:16:06Z

Do I need to clean up the history (squashing, presumably) of this PR? It is a bit messy.

@rustbot review

Mark-Simulacrum · 2022-10-01T20:22:12Z

r=me with commits squashed, thanks!

This PR adds support for fetching version information from the `git-commit-info` file when building the compiler from a source tarball.

jyn514 · 2022-10-01T22:08:23Z

@bors r=Mark-Simulacrum

bors · 2022-10-01T22:08:24Z

📌 Commit fdb3955 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

bors · 2022-10-02T20:55:20Z

⌛ Testing commit fdb3955 with merge de692f1...

bors · 2022-10-02T23:47:41Z

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing de692f1 to master...

rust-timer · 2022-10-03T01:09:42Z

Finished benchmarking commit (de692f1): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean¹	range	count²
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	4.9%	[4.9%, 4.9%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.3%	[-4.7%, -1.1%]	3
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

the arithmetic mean of the percent change ↩
number of relevant changes ↩

Mark-Simulacrum · 2022-10-03T02:41:21Z

src/bootstrap/dist.rs

@@ -897,12 +898,12 @@ impl Step for PlainSourceTarball {

        // Create the version file
        builder.create(&plain_dst_src.join("version"), &builder.rust_version());
-        if let Some(sha) = builder.rust_sha() {
-            builder.create(&plain_dst_src.join("git-commit-hash"), &sha);


Yeah, I overlooked this edit. I think we'll want to keep the git-commit-hash file around; it's consumed by build-manifest (

rust/src/tools/build-manifest/src/versions.rs

Line 144 in 744e397

Some("git-commit-hash") => dest = &mut git_commit,

).

We can probably also fix that code, but I think it's best to leave the file in place.

Oh, my bad, I didn't realize other parts of code used it. Should I open a PR to add that back, or should that be handled somehow else?

If you could open a PR that would be great :)

#102610 is now open :)

…rk-Simulacrum re-add git-commit-hash file to tarballs rust-lang#100557 removed the `git-commit-hash` file and replaced it with `git-commit-info`. However, build-manifest relies on the `git-commit-hash` file being present, so this adds it back. r? `@Mark-Simulacrum`

…crum re-add git-commit-hash file to tarballs rust-lang/rust#100557 removed the `git-commit-hash` file and replaced it with `git-commit-info`. However, build-manifest relies on the `git-commit-hash` file being present, so this adds it back. r? `@Mark-Simulacrum`

rust-highfive assigned Mark-Simulacrum Aug 15, 2022

rustbot added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Aug 15, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 15, 2022

rust-highfive assigned jyn514 and unassigned Mark-Simulacrum Aug 15, 2022

This comment has been minimized.

Sign in to view

jyn514 added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 15, 2022

Zapeth mentioned this pull request Aug 15, 2022

rust: Add release description msys2/MINGW-packages#12570

Merged

This comment has been minimized.

Sign in to view

jyn514 suggested changes Aug 27, 2022

View reviewed changes

jyn514 assigned Mark-Simulacrum Sep 12, 2022

jyn514 approved these changes Sep 12, 2022

View reviewed changes

Mark-Simulacrum reviewed Sep 17, 2022

View reviewed changes

config.toml.example Outdated Show resolved Hide resolved

Mark-Simulacrum reviewed Sep 17, 2022

View reviewed changes

src/bootstrap/tarball.rs Outdated Show resolved Hide resolved

Mark-Simulacrum unassigned jyn514 Sep 17, 2022

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 17, 2022

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 29, 2022

fix: use git-commit-info for version information

fdb3955

This PR adds support for fetching version information from the `git-commit-info` file when building the compiler from a source tarball.

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 1, 2022

bors added the merged-by-bors This PR was explicitly merged by bors. label Oct 2, 2022

bors merged commit de692f1 into rust-lang:master Oct 2, 2022

rustbot added this to the 1.66.0 milestone Oct 2, 2022

bors mentioned this pull request Oct 3, 2022

Support downloading bootstrap from CI in shell scripts #102282

Closed

dawnofmidnight deleted the tarball-commit-info branch October 3, 2022 01:27

Mark-Simulacrum reviewed Oct 3, 2022

View reviewed changes

dawnofmidnight mentioned this pull request Oct 3, 2022

re-add git-commit-hash file to tarballs #102610

Merged

fix: use git-commit-info for version information #100557

fix: use git-commit-info for version information #100557

Conversation

dawnofmidnight commented Aug 15, 2022 • edited Loading

Testing

rust-highfive commented Aug 15, 2022

dawnofmidnight commented Aug 15, 2022

Mark-Simulacrum commented Aug 15, 2022

This comment has been minimized.

dawnofmidnight commented Aug 15, 2022 • edited Loading

jyn514 commented Aug 15, 2022 • edited Loading

Mark-Simulacrum commented Aug 15, 2022

jyn514 commented Aug 15, 2022

Zapeth commented Aug 15, 2022 • edited Loading

jyn514 commented Aug 15, 2022

dawnofmidnight commented Aug 15, 2022 • edited Loading

jyn514 commented Aug 15, 2022

dawnofmidnight commented Aug 15, 2022 • edited Loading

bors commented Aug 21, 2022

dawnofmidnight commented Aug 23, 2022 • edited Loading

jyn514 commented Aug 24, 2022

dawnofmidnight commented Aug 24, 2022

This comment has been minimized.

dawnofmidnight commented Aug 24, 2022

rustbot commented Aug 24, 2022

dawnofmidnight commented Aug 24, 2022 • edited Loading

jyn514 left a comment

Choose a reason for hiding this comment

jyn514 left a comment

Choose a reason for hiding this comment

Mark-Simulacrum commented Sep 17, 2022

dawnofmidnight commented Sep 29, 2022 • edited Loading

Mark-Simulacrum commented Oct 1, 2022

jyn514 commented Oct 1, 2022

bors commented Oct 1, 2022

bors commented Oct 2, 2022

bors commented Oct 2, 2022

rust-timer commented Oct 3, 2022

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Footnotes

Mark-Simulacrum Oct 3, 2022

Choose a reason for hiding this comment

dawnofmidnight Oct 3, 2022 • edited Loading

Choose a reason for hiding this comment

jyn514 Oct 3, 2022

Choose a reason for hiding this comment

dawnofmidnight Oct 3, 2022

Choose a reason for hiding this comment

dawnofmidnight commented Aug 15, 2022 •

edited

Loading

dawnofmidnight commented Aug 15, 2022 •

edited

Loading

jyn514 commented Aug 15, 2022 •

edited

Loading

Zapeth commented Aug 15, 2022 •

edited

Loading

dawnofmidnight commented Aug 15, 2022 •

edited

Loading

dawnofmidnight commented Aug 15, 2022 •

edited

Loading

dawnofmidnight commented Aug 23, 2022 •

edited

Loading

dawnofmidnight commented Aug 24, 2022 •

edited

Loading

dawnofmidnight commented Sep 29, 2022 •

edited

Loading

dawnofmidnight Oct 3, 2022 •

edited

Loading