rust-dev LLVM artifacts are corrupt on FreeBSD 13

[m-ou-se]

On a fresh (x86-64) FreeBSD installation, running ./x.py build library/std results in a segfault:

# ./x.py build library/std
Updating only changed submodules
  Submodules updated in 0.01 seconds
extracting /root/rust/build/cache/2022-04-05/rust-std-beta-x86_64-unknown-freebsd.tar.xz
extracting /root/rust/build/cache/2022-04-05/rustc-beta-x86_64-unknown-freebsd.tar.xz
extracting /root/rust/build/cache/2022-04-05/cargo-beta-x86_64-unknown-freebsd.tar.xz
extracting /root/rust/build/cache/2022-04-05/rustfmt-nightly-x86_64-unknown-freebsd.tar.xz
Building rustbuild

[...]

    Finished dev [unoptimized] target(s) in 16.78s
Building stage0 std artifacts (x86_64-unknown-freebsd -> x86_64-unknown-freebsd)

[...]

   Compiling test v0.0.0 (/root/rust/library/test)
    Finished release [optimized] target(s) in 24.57s
Copying stage0 std from stage0 (x86_64-unknown-freebsd -> x86_64-unknown-freebsd / x86_64-unknown-freebsd)
extracting /root/rust/build/cache/llvm-1388b38c52d1ca9fbc80bf42fa007504fb0b1b41-false/rust-dev-nightly-x86_64-unknown-freebsd.tar.xz to /root/rust/build/x86_64-unknown-freebsd/ci-llvm
Building stage0 compiler artifacts (x86_64-unknown-freebsd -> x86_64-unknown-freebsd)

[...]

   Compiling rustc-main v0.0.0 (/root/rust/compiler/rustc)
    Finished release [optimized] target(s) in 1m 41s
Copying stage0 rustc from stage0 (x86_64-unknown-freebsd -> x86_64-unknown-freebsd / x86_64-unknown-freebsd)
Assembling stage1 compiler (x86_64-unknown-freebsd)
Building stage1 std artifacts (x86_64-unknown-freebsd -> x86_64-unknown-freebsd)
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/root/rust/build/bootstrap/debug/rustc - --crate-name ___ --print=file-names -Csymbol-mangling-version=legacy -Zunstable-options -Zunstable-options '--check-cfg=names()' '--check-cfg=values()' '--check-cfg=values(bootstrap)' '--check-cfg=values(stdarch_intel_sde)' '--check-cfg=values(no_fp_fmt_parse)' '--check-cfg=values(no_global_oom_handling)' '--check-cfg=values(freebsd12)' '--check-cfg=values(backtrace_in_libstd)' '--check-cfg=values(target_env,"libnx")' '--check-cfg=values(target_os,"watchos")' '--check-cfg=values(target_arch,"asmjs","spirv","nvptx","nvptx64","le32","xtensa")' '--check-cfg=values(dont_compile_me)' -Zmacro-backtrace -Clink-args=-Wl,-z,origin '-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Csplit-debuginfo=off -Cprefer-dynamic -Cembed-bitcode=yes '-Zcrate-attr=doc(html_root_url="https://doc.rust-lang.org/nightly/")' --target x86_64-unknown-freebsd --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 254)
  --- stderr
  rustc exited with signal: 11 (core dumped)
Build completed unsuccessfully in 0:02:46

[jyn514]

@m-ou-se it looks like the generated compiler has some codegen issue - does build/x86_64-unknown-freebsd/stage1/bin/rustc --version work?

[m-ou-se]

Yeah that works. That shows rustc 1.62.0-dev as expected.

[m-ou-se]

Running it with --print cfg instead of --version segfaults though.

[jyn514]

Ok. I think it's unlikely to be a build system issue - maybe some target-specific codegen is buggy? I'm not sure how to help further.

@rustbot label -A-rustbuild

[koobs]

Ping @t6 @pkubaj @MikaelUrankar

[Amanieu]

cc @asomers

[asomers]

I can reproduce the failure locally. Unfortunately, the core file isn't much help. This is what gdb shows. Perhaps something is clobbering the stack? Do you know the last working revision? We could try to bisect it.

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000003e800001000 in ?? ()
#2  0x0000000000000000 in ?? ()

[asomers]

If I set download-ci-llvm=false in config.toml, then ./x.py build library/std succeeds, and I can successfully run the binary at build/x86_64-unknown-freebsd/stage1/bin/rustc, with either the --version or --print cfg options. And I can do that on either 13.0-RELEASE or 14.0-CURRENT. And I don't know very much about the toolchain's build process, so I'm kind of stuck right now. Unless somebody can help me find a way to build the broken compiler, or @m-ou-se can post the stack trace from his failing build, I don't think I can help here.

[jyn514]

Ah! It's quite possible the prebuilt llvm artifacts are broken, I don't think we've ever tested them on freebsd. @m-ou-se can you try disabling download-ci-llvm and see if it fixes the issue?

@rustbot label +A-rustbuild +A-llvm

[m-ou-se]

That seems to work, except for #96643. With #96648 and download-ci-llvm = false, it works!

(Some std tests fail to compile though. But that's unrelated.)

[asomers]

So do we need to regenerate the artifacts, and if so how?

[jyn514]

@asomers the artifacts are generated here and then uploaded in CI'

rust/src/bootstrap/dist.rs

Line 2035 in a7d6768

let mut tarball = Tarball::new(builder, "rust-dev", &target.triple);

I don't know why they would be broken only on freebsd. You could try running that dist step locally and using it in a local toolchain.

[asomers]

But I mean from the command line. What's the command to regenerate them?

[m-ou-se]

#96742 removes the broken parts of std. With that patch, all tests pass again on freebsd.

[jyn514]

But I mean from the command line. What's the command to regenerate them?

There is not a simple way to do this. CI is running x dist rust-dev (essentially, along with other components), but I don't know how to replicate the way the archive gets uploaded than redownloaded. It might be as simple as extracting the archive it creates and putting it somewhere in build/freebsd/ci-llvm, but I don't know.

[the8472]

Have you tried applying #91793? That might fix the broken parts.

[jyn514]

@the8472 that's fixing the network APIs - rustc doesn't ever touch the network, so that's unlikely to be the issue.

[the8472]

Sorry, I was too concise, I meant the broken parts of std