Auto merge of rust-lang#109241 - saethlin:miri, r=oli-obk

update Miri

r? `@oli-obk`

src/tools/miri/.github/workflows/ci.yml

@@ -54,8 +54,8 @@ jobs:
             # contains package information of crates installed via `cargo install`.
             ~/.cargo/.crates.toml
             ~/.cargo/.crates2.json
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
-          restore-keys: ${{ runner.os }}-cargo
+          key: ${{ runner.os }}-cargo-reset20230315-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-cargo-reset20230315
 
       - name: Install rustup-toolchain-install-master
         if: ${{ steps.cache.outputs.cache-hit != 'true' }}
@@ -106,8 +106,8 @@ jobs:
             # contains package information of crates installed via `cargo install`.
             ~/.cargo/.crates.toml
             ~/.cargo/.crates2.json
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
-          restore-keys: ${{ runner.os }}-cargo
+          key: ${{ runner.os }}-cargo-reset20230315-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-cargo-reset20230315
 
       - name: Install rustup-toolchain-install-master
         if: ${{ steps.cache.outputs.cache-hit != 'true' }}

src/tools/miri/CONTRIBUTING.md

@@ -129,18 +129,15 @@ development version of Miri using
 ./miri install
 ```
 
-and then you can use it as if it was installed by `rustup`.  Make sure you use
-the same toolchain when calling `cargo miri` that you used when installing Miri!
-Usually this means you have to write `cargo +miri miri ...` to select the `miri`
-toolchain that was installed by `./miri toolchain`.
+and then you can use it as if it was installed by `rustup` as a component of the
+`miri` toolchain. Note that the `miri` and `cargo-miri` executables are placed
+in the `miri` toolchain's sysroot to prevent conflicts with other toolchains.
+The Miri binaries in the `cargo` bin directory (usually `~/.cargo/bin`) are managed by rustup.
 
 There's a test for the cargo wrapper in the `test-cargo-miri` directory; run
 `./run-test.py` in there to execute it. Like `./miri test`, this respects the
 `MIRI_TEST_TARGET` environment variable to execute the test for another target.
 
-Note that installing Miri like this will "take away" Miri management from `rustup`.
-If you want to later go back to a rustup-installed Miri, run `rustup update`.
-
 ### Using a modified standard library
 
 Miri re-builds the standard library into a custom sysroot, so it is fairly easy

src/tools/miri/README.md

@@ -15,6 +15,8 @@ for example:
   or an invalid enum discriminant)
 * **Experimental**: Violations of the [Stacked Borrows] rules governing aliasing
   for reference types
+* **Experimental**: Violations of the Tree Borrows aliasing rules, as an optional
+  alternative to [Stacked Borrows]
 * **Experimental**: Data races
 
 On top of that, Miri will also tell you about memory leaks: when there is memory
@@ -225,6 +227,26 @@ degree documented below):
   reduced feature set. We might ship Miri with a nightly even when some features
   on these targets regress.
 
+### Running tests in parallel
+
+Though it implements Rust threading, Miri itself is a single-threaded interpreter.
+This means that when running `cargo miri test`, you will probably see a dramatic
+increase in the amount of time it takes to run your whole test suite due to the
+inherent interpreter slowdown and a loss of parallelism.
+
+You can get your test suite's parallelism back by running `cargo miri nextest run -jN`
+(note that you will need [`cargo-nextest`](https://nexte.st) installed).
+This works because `cargo-nextest` collects a list of all tests then launches a
+separate `cargo miri run` for each test. You will need to specify a `-j` or `--test-threads`;
+by default `cargo miri nextest run` runs one test at a time. For more details, see the
+[`cargo-nextest` Miri documentation](https://nexte.st/book/miri.html).
+
+Note: This one-test-per-process model means that `cargo miri test` is able to detect data
+races where two tests race on a shared resource, but `cargo miri nextest run` will not detect
+such races.
+
+Note: `cargo-nextest` does not support doctests, see https://github.com/nextest-rs/nextest/issues/16
+
 ### Common Problems
 
 When using the above instructions, you may encounter a number of confusing compiler
@@ -337,9 +359,11 @@ to Miri failing to detect cases of undefined behavior in a program.
 * `-Zmiri-disable-data-race-detector` disables checking for data races.  Using
   this flag is **unsound**. This implies `-Zmiri-disable-weak-memory-emulation`.
 * `-Zmiri-disable-stacked-borrows` disables checking the experimental
-  [Stacked Borrows] aliasing rules.  This can make Miri run faster, but it also
-  means no aliasing violations will be detected.  Using this flag is **unsound**
-  (but the affected soundness rules are experimental).
+  aliasing rules to track borrows ([Stacked Borrows] and Tree Borrows).
+  This can make Miri run faster, but it also means no aliasing violations will
+  be detected. Using this flag is **unsound** (but the affected soundness rules
+  are experimental). Later flags take precedence: borrow tracking can be reactivated
+  by `-Zmiri-tree-borrows`.
 * `-Zmiri-disable-validation` disables enforcing validity invariants, which are
   enforced by default.  This is mostly useful to focus on other failures (such
   as out-of-bounds accesses) first.  Setting this flag means Miri can miss bugs
@@ -401,6 +425,9 @@ to Miri failing to detect cases of undefined behavior in a program.
 * `-Zmiri-track-weak-memory-loads` shows a backtrace when weak memory emulation returns an outdated
   value from a load. This can help diagnose problems that disappear under
   `-Zmiri-disable-weak-memory-emulation`.
+* `-Zmiri-tree-borrows` replaces [Stacked Borrows] with the Tree Borrows rules.
+  The soundness rules are already experimental without this flag, but even more
+  so with this flag.
 * `-Zmiri-force-page-size=<num>` overrides the default page size for an architecture, in multiples of 1k.
   `4` is default for most targets. This value should always be a power of 2 and nonzero.
 
@@ -415,7 +442,7 @@ Some native rustc `-Z` flags are also very relevant for Miri:
   functions.  This is needed so that Miri can execute such functions, so Miri
   sets this flag per default.
 * `-Zmir-emit-retag` controls whether `Retag` statements are emitted. Miri
-  enables this per default because it is needed for [Stacked Borrows].
+  enables this per default because it is needed for [Stacked Borrows] and Tree Borrows.
 
 Moreover, Miri recognizes some environment variables:
 
@@ -481,120 +508,8 @@ binaries, and as such worth documenting:
 ## Miri `extern` functions
 
 Miri provides some `extern` functions that programs can import to access
-Miri-specific functionality:
-
-```rust
-#[cfg(miri)]
-extern "Rust" {
-    /// Miri-provided extern function to mark the block `ptr` points to as a "root"
-    /// for some static memory. This memory and everything reachable by it is not
-    /// considered leaking even if it still exists when the program terminates.
-    ///
-    /// `ptr` has to point to the beginning of an allocated block.
-    fn miri_static_root(ptr: *const u8);
-
-    // Miri-provided extern function to get the amount of frames in the current backtrace.
-    // The `flags` argument must be `0`.
-    fn miri_backtrace_size(flags: u64) -> usize;
-
-    /// Miri-provided extern function to obtain a backtrace of the current call stack.
-    /// This writes a slice of pointers into `buf` - each pointer is an opaque value
-    /// that is only useful when passed to `miri_resolve_frame`.
-    /// `buf` must have `miri_backtrace_size(0) * pointer_size` bytes of space.
-    /// The `flags` argument must be `1`.
-    fn miri_get_backtrace(flags: u64, buf: *mut *mut ());
-
-    /// Miri-provided extern function to resolve a frame pointer obtained
-    /// from `miri_get_backtrace`. The `flags` argument must be `1`,
-    /// and `MiriFrame` should be declared as follows:
-    ///
-    /// ```rust
-    /// #[repr(C)]
-    /// struct MiriFrame {
-    ///     // The size of the name of the function being executed, encoded in UTF-8
-    ///     name_len: usize,
-    ///     // The size of filename of the function being executed, encoded in UTF-8
-    ///     filename_len: usize,
-    ///     // The line number currently being executed in `filename`, starting from '1'.
-    ///     lineno: u32,
-    ///     // The column number currently being executed in `filename`, starting from '1'.
-    ///     colno: u32,
-    ///     // The function pointer to the function currently being executed.
-    ///     // This can be compared against function pointers obtained by
-    ///     // casting a function (e.g. `my_fn as *mut ()`)
-    ///     fn_ptr: *mut ()
-    /// }
-    /// ```
-    ///
-    /// The fields must be declared in exactly the same order as they appear in `MiriFrame` above.
-    /// This function can be called on any thread (not just the one which obtained `frame`).
-    fn miri_resolve_frame(frame: *mut (), flags: u64) -> MiriFrame;
-
-    /// Miri-provided extern function to get the name and filename of the frame provided by `miri_resolve_frame`.
-    /// `name_buf` and `filename_buf` should be allocated with the `name_len` and `filename_len` fields of `MiriFrame`.
-    /// The flags argument must be `0`.
-    fn miri_resolve_frame_names(ptr: *mut (), flags: u64, name_buf: *mut u8, filename_buf: *mut u8);
-
-    /// Miri-provided extern function to begin unwinding with the given payload.
-    ///
-    /// This is internal and unstable and should not be used; we give it here
-    /// just to be complete.
-    fn miri_start_panic(payload: *mut u8) -> !;
-
-    /// Miri-provided extern function to get the internal unique identifier for the allocation that a pointer
-    /// points to. If this pointer is invalid (not pointing to an allocation), interpretation will abort.
-    ///
-    /// This is only useful as an input to `miri_print_borrow_stacks`, and it is a separate call because
-    /// getting a pointer to an allocation at runtime can change the borrow stacks in the allocation.
-    /// This function should be considered unstable. It exists only to support `miri_print_borrow_stacks` and so
-    /// inherits all of its instability.
-    fn miri_get_alloc_id(ptr: *const ()) -> u64;
-
-    /// Miri-provided extern function to print (from the interpreter, not the program) the contents of all
-    /// borrow stacks in an allocation. The leftmost tag is the bottom of the stack.
-    /// The format of what this emits is unstable and may change at any time. In particular, users should be
-    /// aware that Miri will periodically attempt to garbage collect the contents of all stacks. Callers of
-    /// this function may wish to pass `-Zmiri-tag-gc=0` to disable the GC.
-    ///
-    /// This function is extremely unstable. At any time the format of its output may change, its signature may
-    /// change, or it may be removed entirely.
-    fn miri_print_borrow_stacks(alloc_id: u64);
-
-    /// Miri-provided extern function to print (from the interpreter, not the
-    /// program) the contents of a section of program memory, as bytes. Bytes
-    /// written using this function will emerge from the interpreter's stdout.
-    fn miri_write_to_stdout(bytes: &[u8]);
-
-    /// Miri-provided extern function to print (from the interpreter, not the
-    /// program) the contents of a section of program memory, as bytes. Bytes
-    /// written using this function will emerge from the interpreter's stderr.
-    fn miri_write_to_stderr(bytes: &[u8]);
-
-    /// Miri-provided extern function to allocate memory from the interpreter.
-    /// 
-    /// This is useful when no fundamental way of allocating memory is
-    /// available, e.g. when using `no_std` + `alloc`.
-    fn miri_alloc(size: usize, align: usize) -> *mut u8;
-
-    /// Miri-provided extern function to deallocate memory.
-    fn miri_dealloc(ptr: *mut u8, size: usize, align: usize);
-
-    /// Convert a path from the host Miri runs on to the target Miri interprets.
-    /// Performs conversion of path separators as needed.
-    ///
-    /// Usually Miri performs this kind of conversion automatically. However, manual conversion
-    /// might be necessary when reading an environment variable that was set on the host
-    /// (such as TMPDIR) and using it as a target path.
-    ///
-    /// Only works with isolation disabled.
-    ///
-    /// `in` must point to a null-terminated string, and will be read as the input host path.
-    /// `out` must point to at least `out_size` many bytes, and the result will be stored there
-    /// with a null terminator.
-    /// Returns 0 if the `out` buffer was large enough, and the required size otherwise.
-    fn miri_host_to_target_path(path: *const std::ffi::c_char, out: *mut std::ffi::c_char, out_size: usize) -> usize;
-}
-```
+Miri-specific functionality. They are declared in
+[/tests/utils/miri\_extern.rs](/tests/utils/miri_extern.rs).
 
 ## Contributing and getting help
 

src/tools/miri/ci.sh

@@ -62,8 +62,8 @@ function run_tests {
   if [ "$HOST_TARGET" = x86_64-unknown-linux-gnu ]; then
     # These act up on Windows (`which miri` produces a filename that does not exist?!?),
     # so let's do this only on Linux. Also makes sure things work without these set.
-    export RUSTC=$(which rustc)
-    export MIRI=$(which miri)
+    export RUSTC=$(which rustc) # Produces a warning unless we also set MIRI
+    export MIRI=$(rustc +miri --print sysroot)/bin/miri
   fi
   mkdir -p .cargo
   echo 'build.rustc-wrapper = "thisdoesnotexist"' > .cargo/config.toml

src/tools/miri/miri

@@ -6,8 +6,8 @@ USAGE=$(cat <<"EOF"
 ./miri install <flags>:
 Installs the miri driver and cargo-miri. <flags> are passed to `cargo
 install`. Sets up the rpath such that the installed binary should work in any
-working directory. However, the rustup toolchain when invoking `cargo miri`
-needs to be the same one used for `./miri install`.
+working directory. Note that the binaries are placed in the `miri` toolchain
+sysroot, to prevent conflicts with other toolchains.
 
 ./miri build <flags>:
 Just build miri. <flags> are passed to `cargo build`.
@@ -281,8 +281,9 @@ find_sysroot() {
 case "$COMMAND" in
 install)
     # "--locked" to respect the Cargo.lock file if it exists.
-    $CARGO install $CARGO_EXTRA_FLAGS --path "$MIRIDIR" --force --locked "$@"
-    $CARGO install $CARGO_EXTRA_FLAGS --path "$MIRIDIR"/cargo-miri --force --locked "$@"
+    # Install binaries to the miri toolchain's sysroot so they do not interact with other toolchains.
+    $CARGO install $CARGO_EXTRA_FLAGS --path "$MIRIDIR" --force --locked --root "$SYSROOT" "$@"
+    $CARGO install $CARGO_EXTRA_FLAGS --path "$MIRIDIR"/cargo-miri --force --locked --root "$SYSROOT" "$@"
     ;;
 check)
     # Check, and let caller control flags.

src/tools/miri/rust-version

@@ -1 +1 @@
-c4e0cd966062ca67daed20775f4e8a60c28e57df
+511364e7874dba9649a264100407e4bffe7b5425

src/tools/miri/src/bin/miri.rs

@@ -32,7 +32,7 @@ use rustc_middle::{
 };
 use rustc_session::{config::CrateType, search_paths::PathKind, CtfeBacktrace};
 
-use miri::{BacktraceStyle, ProvenanceMode, RetagFields};
+use miri::{BacktraceStyle, BorrowTrackerMethod, ProvenanceMode, RetagFields};
 
 struct MiriCompilerCalls {
     miri_config: miri::MiriConfig,
@@ -317,6 +317,8 @@ fn main() {
             miri_config.validate = false;
         } else if arg == "-Zmiri-disable-stacked-borrows" {
             miri_config.borrow_tracker = None;
+        } else if arg == "-Zmiri-tree-borrows" {
+            miri_config.borrow_tracker = Some(BorrowTrackerMethod::TreeBorrows);
         } else if arg == "-Zmiri-disable-data-race-detector" {
             miri_config.data_race_detector = false;
             miri_config.weak_memory_emulation = false;