Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible data race in Rust runtime #780

Closed
daviddrysdale opened this issue Mar 27, 2020 · 7 comments
Closed

Possible data race in Rust runtime #780

daviddrysdale opened this issue Mar 27, 2020 · 7 comments
Assignees
@blaxill

Comments

@daviddrysdale
Copy link
Contributor

On trying a test run under Thread Sanitizer (cf #778) there's a possible data race between wait_on_channels by the logging node, and channel_write from the Wasm node.

% export TSAN_OPTIONS="halt_on_error=1 report_atomic_races=0"
% RUSTFLAGS="-Z sanitizer=thread" cargo test --manifest-path=./examples/abitest/tests/Cargo.toml --target x86_64-unknown-linux-gnu -- --nocapture
    Finished test [unoptimized + debuginfo] target(s) in 0.09s
     Running target/x86_64-unknown-linux-gnu/debug/deps/abitest_tests-70de4023c24259b6

running 1 test
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Z sanitizer=thread --target wasm32-unknown-unknown --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
--- stdout
___.wasm
lib___.rlib
___.wasm
lib___.a
/usr/local/google/home/drysdale/.rustup/toolchains/nightly-2020-02-06-x86_64-unknown-linux-gnu
debug_assertions
proc_macro
sanitize="thread"
target_arch="wasm32"
target_endian="little"
target_env=""
target_has_atomic="16"
target_has_atomic="32"
target_has_atomic="64"
target_has_atomic="8"
target_has_atomic="ptr"
target_has_atomic_load_store="16"
target_has_atomic_load_store="32"
target_has_atomic_load_store="64"
target_has_atomic_load_store="8"
target_has_atomic_load_store="ptr"
target_os="unknown"
target_pointer_width="32"
target_thread_local
target_vendor="unknown"

--- stderr
error: ThreadSanitizer only works with the `x86_64-unknown-linux-gnu` or `x86_64-apple-darwin` target

warning: dropping unsupported crate type `dylib` for target `wasm32-unknown-unknown`

warning: dropping unsupported crate type `proc-macro` for target `wasm32-unknown-unknown`

error: aborting due to previous error


error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Z sanitizer=thread --target wasm32-unknown-unknown --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
--- stdout
___.wasm
lib___.rlib
___.wasm
lib___.a
/usr/local/google/home/drysdale/.rustup/toolchains/nightly-2020-02-06-x86_64-unknown-linux-gnu
debug_assertions
proc_macro
sanitize="thread"
target_arch="wasm32"
target_endian="little"
target_env=""
target_has_atomic="16"
target_has_atomic="32"
target_has_atomic="64"
target_has_atomic="8"
target_has_atomic="ptr"
target_has_atomic_load_store="16"
target_has_atomic_load_store="32"
target_has_atomic_load_store="64"
target_has_atomic_load_store="8"
target_has_atomic_load_store="ptr"
target_os="unknown"
target_pointer_width="32"
target_thread_local
target_vendor="unknown"

--- stderr
error: ThreadSanitizer only works with the `x86_64-unknown-linux-gnu` or `x86_64-apple-darwin` target

warning: dropping unsupported crate type `dylib` for target `wasm32-unknown-unknown`

warning: dropping unsupported crate type `proc-macro` for target `wasm32-unknown-unknown`

error: aborting due to previous error


2020-03-27 15:04:46,664 DEBUG [oak_runtime::node::wasm] new_instance: discovering frontend-config frontend_oak_main
2020-03-27 15:04:47,633 DEBUG [oak_runtime::node::wasm] new_instance: starting frontend-config frontend_oak_main
2020-03-27 15:04:47,635 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:47,735 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:47,836 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:47,937 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,037 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,138 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,239 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,340 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,440 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,541 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,642 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,647 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047240), I32(1047248)])
2020-03-27 15:04:48,647 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047240 1047248
2020-03-27 15:04:48,649 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 0 RuntimeArgs([I32(1049284), I32(14), I32(1395328), I32(8), I64(-5089638740390981732)])
2020-03-27 15:04:48,649 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create: 1049284 14 1395328 8 13357105333318569884
2020-03-27 15:04:48,649 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create config_name is: logging-config
2020-03-27 15:04:48,650 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create entrypoint is: oak_main
2020-03-27 15:04:48,652 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(-5089638740390981732)])
2020-03-27 15:04:48,652 DEBUG [oak_runtime::runtime] wait_on_channels: channels not ready, parking thread Thread { id: ThreadId(4), name: None }
2020-03-27 15:04:48,652 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 13357105333318569884
2020-03-27 15:04:48,699 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:48,699 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 0 RuntimeArgs([I32(1049428), I32(14), I32(1049442), I32(16), I64(1791648940387253873)])
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create: 1049428 14 1049442 16 1791648940387253873
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create config_name is: backend-config
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create entrypoint is: backend_oak_main
2020-03-27 15:04:48,701 DEBUG [oak_runtime::node::wasm] new_instance: discovering backend-config backend_oak_main
2020-03-27 15:04:48,742 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,843 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:48,944 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,044 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,145 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,245 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,346 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,442 DEBUG [oak_runtime::node::wasm] new_instance: starting backend-config backend_oak_main
2020-03-27 15:04:49,444 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(1791648940387253873)])
2020-03-27 15:04:49,445 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 1791648940387253873
2020-03-27 15:04:49,446 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:49,446 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:49,446 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,460 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 4 RuntimeArgs([I64(1676672801125846535), I32(1049552), I32(0), I32(1507416), I32(1)])
2020-03-27 15:04:49,460 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_write: 1676672801125846535 1049552 0 1507416 1
2020-03-27 15:04:49,467 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(-2987603100449681604)])
2020-03-27 15:04:49,467 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 15459140973259870012
2020-03-27 15:04:49,471 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:49,472 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:49,473 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 0 RuntimeArgs([I32(1049428), I32(14), I32(1049442), I32(16), I64(-6684026489260364187)])
2020-03-27 15:04:49,474 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create: 1049428 14 1049442 16 11762717584449187429
2020-03-27 15:04:49,474 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create config_name is: backend-config
2020-03-27 15:04:49,474 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create entrypoint is: backend_oak_main
2020-03-27 15:04:49,474 DEBUG [oak_runtime::node::wasm] new_instance: discovering backend-config backend_oak_main
2020-03-27 15:04:49,547 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,647 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,748 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,849 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:49,949 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,050 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,137 DEBUG [oak_runtime::node::wasm] new_instance: starting backend-config backend_oak_main
2020-03-27 15:04:50,139 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(-6684026489260364187)])
2020-03-27 15:04:50,139 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 11762717584449187429
2020-03-27 15:04:50,141 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:50,141 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:50,151 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,152 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 3 RuntimeArgs([I32(1046680), I32(1046688)])
2020-03-27 15:04:50,152 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: channel_create: 1046680 1046688
2020-03-27 15:04:50,154 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 0 RuntimeArgs([I32(1050108), I32(14), I32(1059780), I32(8), I64(2293680208635920034)])
2020-03-27 15:04:50,154 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: node_create: 1050108 14 1059780 8 2293680208635920034
2020-03-27 15:04:50,155 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: node_create config_name is: logging-config
2020-03-27 15:04:50,155 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: node_create entrypoint is: oak_main
2020-03-27 15:04:50,155 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 4 RuntimeArgs([I64(1277197695412101525), I32(1049552), I32(0), I32(1507416), I32(1)])
2020-03-27 15:04:50,156 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_write: 1277197695412101525 1049552 0 1507416 1
2020-03-27 15:04:50,162 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 2 RuntimeArgs([I64(-8944433438557824175)])
2020-03-27 15:04:50,163 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_close: 9502310635151727441
2020-03-27 15:04:50,164 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 2 RuntimeArgs([I64(2293680208635920034)])
2020-03-27 15:04:50,164 DEBUG [oak_runtime::runtime] wait_on_channels: channels not ready, parking thread Thread { id: ThreadId(7), name: None }
2020-03-27 15:04:50,165 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: channel_close: 2293680208635920034
2020-03-27 15:04:50,167 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 3 RuntimeArgs([I32(1047368), I32(1047376)])
2020-03-27 15:04:50,167 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: channel_create: 1047368 1047376
2020-03-27 15:04:50,169 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: invoke_index: 0 RuntimeArgs([I32(1049428), I32(14), I32(1049442), I32(16), I64(6553477005041356551)])
2020-03-27 15:04:50,169 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create: 1049428 14 1049442 16 6553477005041356551
2020-03-27 15:04:50,169 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create config_name is: backend-config
2020-03-27 15:04:50,169 DEBUG [oak_runtime::node::wasm] frontend-config-Thread { id: ThreadId(3), name: None }: node_create entrypoint is: backend_oak_main
2020-03-27 15:04:50,170 DEBUG [oak_runtime::node::wasm] new_instance: discovering backend-config backend_oak_main
2020-03-27 15:04:50,219 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 5 RuntimeArgs([I64(7374366863468297640), I32(1179672), I32(2), I32(1046604), I32(1179712), I32(2), I32(1046608)])
2020-03-27 15:04:50,220 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: channel_read: 7374366863468297640 1179672 2 1046604 1179712 2 1046608
2020-03-27 15:04:50,251 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,352 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 4 RuntimeArgs([I64(-3905051140228563756), I32(1179904), I32(163), I32(1), I32(0)])
2020-03-27 15:04:50,352 INFO  [oak_tests] no pending gRPC response message; poll again soon
2020-03-27 15:04:50,352 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: channel_write: 14541692933480987860 1179904 163 1 0
2020-03-27 15:04:50,352 DEBUG [oak_runtime::runtime] wait_on_channels: thread Thread { id: ThreadId(7), name: None } re-woken
2020-03-27 15:04:50,415 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: invoke_index: 6 RuntimeArgs([I32(1179728), I32(1)])
2020-03-27 15:04:50,415 DEBUG [oak_runtime::node::wasm] backend-config-Thread { id: ThreadId(5), name: None }: wait_on_channels: 1179728 1
2020-03-27 15:04:50,415 DEBUG [oak_runtime::runtime] wait_on_channels: channels not ready, parking thread Thread { id: ThreadId(5), name: None }
==================
WARNING: ThreadSanitizer: data race (pid=233640)
  Write of size 8 at 0x7b0800070980 by thread T6:
    #0 free /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:706 (abitest_tests-70de4023c24259b6+0x44d48)
    #1 alloc::alloc::dealloc /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/alloc.rs:103 (abitest_tests-70de4023c24259b6+0x233715)
    #2 <alloc::alloc::Global as core::alloc::AllocRef>::dealloc /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/alloc.rs:174 (abitest_tests-70de4023c24259b6+0x2339e8)
    #3 alloc::sync::Arc<T>::drop_slow /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/sync.rs:743 (abitest_tests-70de4023c24259b6+0x158767)
    #4 <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/sync.rs:1249 (abitest_tests-70de4023c24259b6+0x158e70)
    #5 core::ptr::drop_in_place /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ptr/mod.rs:174 (abitest_tests-70de4023c24259b6+0x16c93d)
    #6 oak_runtime::runtime::Runtime::wait_on_channels /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:259 (abitest_tests-70de4023c24259b6+0x147d58)
    #7 oak_runtime::node::logger::logger /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/logger.rs:84 (abitest_tests-70de4023c24259b6+0x24a439)
    #8 <oak_runtime::node::logger::LogNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/logger.rs:60 (abitest_tests-70de4023c24259b6+0x24781c)
    #9 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204c40)
    #10 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba704)
    #11 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9514)
    #12 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd23f)
    #13 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #14 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d9684)
    #15 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba0b0)
    #16 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16630d)
    #17 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

  Previous atomic write of size 8 at 0x7b0800070980 by thread T4 (mutexes: read M17, write M53):
    #0 __tsan_atomic64_fetch_sub /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:647 (abitest_tests-70de4023c24259b6+0x885c1)
    #1 core::sync::atomic::atomic_sub /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/sync/atomic.rs:2314 (abitest_tests-70de4023c24259b6+0x6900b9)
    #2 core::sync::atomic::AtomicUsize::fetch_sub /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/sync/atomic.rs:1654 (abitest_tests-70de4023c24259b6+0x1bb109)
    #3 <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/sync.rs:1214 (abitest_tests-70de4023c24259b6+0x158e2f)
    #4 core::ptr::drop_in_place /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ptr/mod.rs:174 (abitest_tests-70de4023c24259b6+0x16c93d)
    #5 oak_runtime::runtime::Runtime::channel_write::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:315 (abitest_tests-70de4023c24259b6+0x16260f)
    #6 oak_runtime::runtime::channel::ChannelMapping::with_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/channel.rs:224 (abitest_tests-70de4023c24259b6+0x136e22)
    #7 oak_runtime::runtime::Runtime::channel_write /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:266 (abitest_tests-70de4023c24259b6+0x147f58)
    #8 oak_runtime::node::wasm::WasmInterface::channel_write /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:315 (abitest_tests-70de4023c24259b6+0x22269e)
    #9 <oak_runtime::node::wasm::WasmInterface as wasmi::host::Externals>::invoke_index /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:609 (abitest_tests-70de4023c24259b6+0x2283cc)
    #10 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:151 (abitest_tests-70de4023c24259b6+0x204872)
    #11 wasmi::runner::Interpreter::run_interpreter_loop /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:307 (abitest_tests-70de4023c24259b6+0x1da649)
    #12 wasmi::runner::Interpreter::start_execution /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:215 (abitest_tests-70de4023c24259b6+0x1d983f)
    #13 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:146 (abitest_tests-70de4023c24259b6+0x204902)
    #14 wasmi::module::ModuleInstance::invoke_export /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/module.rs:639 (abitest_tests-70de4023c24259b6+0x21cecc)
    #15 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:841 (abitest_tests-70de4023c24259b6+0x1f151d)
    #16 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204bd0)
    #17 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba774)
    #18 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9584)
    #19 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd192)
    #20 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #21 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d95fa)
    #22 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba490)
    #23 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16636d)
    #24 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

  Mutex M17 (0x7b1400006360) created at:
    #0 pthread_rwlock_wrlock /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1366 (abitest_tests-70de4023c24259b6+0x48596)
    #1 std::sys::unix::rwlock::RWLock::write /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys/unix/rwlock.rs:71 (abitest_tests-70de4023c24259b6+0x247311)
    #2 std::sys_common::rwlock::RWLock::write /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/rwlock.rs:48 (abitest_tests-70de4023c24259b6+0x204ae9)
    #3 std::sync::rwlock::RwLock<T>::write /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sync/rwlock.rs:267 (abitest_tests-70de4023c24259b6+0x1fbcab)
    #4 oak_runtime::runtime::channel::ChannelMapping::new_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/channel.rs:163 (abitest_tests-70de4023c24259b6+0x140d07)
    #5 oak_runtime::runtime::Runtime::new_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:179 (abitest_tests-70de4023c24259b6+0x146fe4)
    #6 oak_runtime::runtime::Runtime::run /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:110 (abitest_tests-70de4023c24259b6+0x14616c)
    #7 oak_runtime::config::configure_and_run /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/config.rs:119 (abitest_tests-70de4023c24259b6+0x1b7f7b)
    #8 abitest_tests::tests::test_abi /usr/local/google/home/drysdale/src/oak/examples/abitest/tests/src/tests.rs:55 (abitest_tests-70de4023c24259b6+0xb937a)
    #9 abitest_tests::tests::test_abi::{{closure}} /usr/local/google/home/drysdale/src/oak/examples/abitest/tests/src/tests.rs:45 (abitest_tests-70de4023c24259b6+0xb6214)
    #10 core::ops::function::FnOnce::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0xb6437)
    #11 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x50533e)

  Mutex M53 (0x7b0c0001cbc0) created at:
    #0 pthread_mutex_init /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1225 (abitest_tests-70de4023c24259b6+0x472dd)
    #1 std::sys::unix::mutex::Mutex::init /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys/unix/mutex.rs:50 (abitest_tests-70de4023c24259b6+0x22f482)
    #2 std::sys_common::mutex::Mutex::init /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/mutex.rs:32 (abitest_tests-70de4023c24259b6+0x1db299)
    #3 std::sync::mutex::Mutex<T>::new /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sync/mutex.rs:173 (abitest_tests-70de4023c24259b6+0x194f83)
    #4 oak_runtime::runtime::channel::Channel::new /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/channel.rs:103 (abitest_tests-70de4023c24259b6+0x1403b3)
    #5 oak_runtime::runtime::channel::ChannelMapping::new_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/channel.rs:164 (abitest_tests-70de4023c24259b6+0x140d8d)
    #6 oak_runtime::runtime::Runtime::new_channel /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:179 (abitest_tests-70de4023c24259b6+0x146fe4)
    #7 oak_runtime::node::wasm::WasmInterface::channel_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:222 (abitest_tests-70de4023c24259b6+0x22188d)
    #8 <oak_runtime::node::wasm::WasmInterface as wasmi::host::Externals>::invoke_index /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:594 (abitest_tests-70de4023c24259b6+0x227489)
    #9 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:151 (abitest_tests-70de4023c24259b6+0x204872)
    #10 wasmi::runner::Interpreter::run_interpreter_loop /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:307 (abitest_tests-70de4023c24259b6+0x1da649)
    #11 wasmi::runner::Interpreter::start_execution /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:215 (abitest_tests-70de4023c24259b6+0x1d983f)
    #12 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:146 (abitest_tests-70de4023c24259b6+0x204902)
    #13 wasmi::module::ModuleInstance::invoke_export /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/module.rs:639 (abitest_tests-70de4023c24259b6+0x21cecc)
    #14 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:841 (abitest_tests-70de4023c24259b6+0x1f151d)
    #15 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204bd0)
    #16 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba774)
    #17 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9584)
    #18 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd192)
    #19 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #20 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d95fa)
    #21 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba490)
    #22 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16636d)
    #23 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

  Thread T6 (tid=234002, running) created by thread T4 at:
    #0 pthread_create /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967 (abitest_tests-70de4023c24259b6+0x45c9b)
    #1 std::sys::unix::thread::Thread::new /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libstd/sys/unix/thread.rs:68 (abitest_tests-70de4023c24259b6+0x970fdd)
    #2 std::thread::Builder::spawn /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:386 (abitest_tests-70de4023c24259b6+0x1ba832)
    #3 std::thread::spawn /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:619 (abitest_tests-70de4023c24259b6+0x1b8ccd)
    #4 <oak_runtime::node::logger::LogNode as oak_runtime::node::Node>::start /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/logger.rs:58 (abitest_tests-70de4023c24259b6+0x249bd4)
    #5 oak_runtime::runtime::RuntimeRef::node_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:487 (abitest_tests-70de4023c24259b6+0x14909e)
    #6 oak_runtime::node::wasm::WasmInterface::node_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:199 (abitest_tests-70de4023c24259b6+0x221422)
    #7 <oak_runtime::node::wasm::WasmInterface as wasmi::host::Externals>::invoke_index /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:536 (abitest_tests-70de4023c24259b6+0x225d36)
    #8 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:151 (abitest_tests-70de4023c24259b6+0x204872)
    #9 wasmi::runner::Interpreter::run_interpreter_loop /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:307 (abitest_tests-70de4023c24259b6+0x1da649)
    #10 wasmi::runner::Interpreter::start_execution /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:215 (abitest_tests-70de4023c24259b6+0x1d983f)
    #11 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:146 (abitest_tests-70de4023c24259b6+0x204902)
    #12 wasmi::module::ModuleInstance::invoke_export /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/module.rs:639 (abitest_tests-70de4023c24259b6+0x21cecc)
    #13 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:841 (abitest_tests-70de4023c24259b6+0x1f151d)
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204bd0)
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba774)
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9584)
    #17 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd192)
    #18 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #19 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d95fa)
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba490)
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16636d)
    #22 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

  Thread T4 (tid=233996, running) created by thread T2 at:
    #0 pthread_create /checkout/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967 (abitest_tests-70de4023c24259b6+0x45c9b)
    #1 std::sys::unix::thread::Thread::new /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libstd/sys/unix/thread.rs:68 (abitest_tests-70de4023c24259b6+0x970fdd)
    #2 std::thread::Builder::spawn /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:386 (abitest_tests-70de4023c24259b6+0x1ba8fa)
    #3 std::thread::spawn /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:619 (abitest_tests-70de4023c24259b6+0x1b8e1d)
    #4 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:824 (abitest_tests-70de4023c24259b6+0x22bc1d)
    #5 oak_runtime::runtime::RuntimeRef::node_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/runtime/mod.rs:487 (abitest_tests-70de4023c24259b6+0x14909e)
    #6 oak_runtime::node::wasm::WasmInterface::node_create /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:199 (abitest_tests-70de4023c24259b6+0x221422)
    #7 <oak_runtime::node::wasm::WasmInterface as wasmi::host::Externals>::invoke_index /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:536 (abitest_tests-70de4023c24259b6+0x225d36)
    #8 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:151 (abitest_tests-70de4023c24259b6+0x204872)
    #9 wasmi::runner::Interpreter::run_interpreter_loop /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:307 (abitest_tests-70de4023c24259b6+0x1da649)
    #10 wasmi::runner::Interpreter::start_execution /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/runner.rs:215 (abitest_tests-70de4023c24259b6+0x1d983f)
    #11 wasmi::func::FuncInstance::invoke /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/func.rs:146 (abitest_tests-70de4023c24259b6+0x204902)
    #12 wasmi::module::ModuleInstance::invoke_export /usr/local/google/home/drysdale/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmi-0.6.2/src/module.rs:639 (abitest_tests-70de4023c24259b6+0x21cecc)
    #13 <oak_runtime::node::wasm::WasmNode as oak_runtime::node::Node>::start::{{closure}} /usr/local/google/home/drysdale/src/oak/oak/server/rust/oak_runtime/src/node/wasm.rs:841 (abitest_tests-70de4023c24259b6+0x1f151d)
    #14 std::sys_common::backtrace::__rust_begin_short_backtrace /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/sys_common/backtrace.rs:129 (abitest_tests-70de4023c24259b6+0x204bd0)
    #15 std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:475 (abitest_tests-70de4023c24259b6+0x1ba774)
    #16 <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:318 (abitest_tests-70de4023c24259b6+0x1d9584)
    #17 std::panicking::try::do_call /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panicking.rs:303 (abitest_tests-70de4023c24259b6+0x1fd192)
    #18 __rust_maybe_catch_panic /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e//src/libpanic_unwind/lib.rs:86 (abitest_tests-70de4023c24259b6+0x972236)
    #19 std::panic::catch_unwind /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/panic.rs:394 (abitest_tests-70de4023c24259b6+0x1d95fa)
    #20 std::thread::Builder::spawn_unchecked::{{closure}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libstd/thread/mod.rs:474 (abitest_tests-70de4023c24259b6+0x1ba490)
    #21 core::ops::function::FnOnce::call_once{{vtable-shim}} /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/libcore/ops/function.rs:232 (abitest_tests-70de4023c24259b6+0x16636d)
    #22 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/boxed.rs:1016 (abitest_tests-70de4023c24259b6+0x95dd7e)

SUMMARY: ThreadSanitizer: data race /rustc/58b834344fc7b9185e7a50db1ff24e5eb07dae5e/src/liballoc/alloc.rs:103 in alloc::alloc::dealloc
==================
error: test failed, to rerun pass '--lib'
drysdale@ring:~/src/oak{master}:
@blaxill
Copy link
Contributor

blaxill commented Mar 27, 2020

@daviddrysdale any tips on how to investigate this further? I have a suspicion this is complaining there is a race on the Arc/Weak that corresponds to WaitingThreads, which could be a false positive.

@daviddrysdale
Copy link
Contributor Author

Yeah, it looks to be related to the various layers of indirection involved in passing around a &Arc<Thread> (a reference to an atomic-reference-counted-pointer, which internally contains another atomic-reference-counted-pointer); interesting in particular that one access is a "write" and one is an "atomic write". Need to dig deeper to figure out whether it's a false positive or not, though.

@blaxill
Copy link
Contributor

blaxill commented Mar 27, 2020

The reference there is unnecessary, that seems to get rid of the data race warning. I think the original complaint was that the reference was being using in a closure and the sanitizer couldn't guarantee this wasn't an issue - not sure. It doesn't seem to mind it with a copy though

However, there are now some potential lock inversion warnings im looking into.

@blaxill blaxill self-assigned this Mar 27, 2020
@blaxill blaxill mentioned this issue Mar 27, 2020
Merged
6 tasks
@blaxill
Copy link
Contributor

blaxill commented Mar 27, 2020

Ok the data race warning is a false positive due to Arc that is fixed in nightlies newer than 3/19 (c.f. rust-lang/rust#65097). Unfortunately there are components missing so updating the Docker image at this time would be a bad idea (clippy and rls are missing for example).

There were some lock inversion warnings that seem legitimate that I have fixed in #784, this can be verified by

  • running RUSTFLAGS="-Z sanitizer=thread" cargo -Zbuild-std test --manifest-path=./examples/abitest/tests/Cargo.toml --target x86_64-unknown-linux-gnu -- --nocapture
    (Note I believe -Zbuild-std, to instrument the standard library, is required).
  • Using a nightly compiler newer than 2020.03.19

and seeing there are no errors/warnings.

@blaxill blaxill mentioned this issue Mar 27, 2020
Closed
@daviddrysdale
Copy link
Contributor Author

BTW, what were the lock inversion warnings? (I didn't spot any inversions (#779) from code-reading, but that's not a particularly reliable way to find them.)

@blaxill
Copy link
Contributor

blaxill commented Mar 30, 2020

The warnings corresponded to one instance of inversion against your list "runtime.channels -> runtime.nodes". One other change in #784 was not strictly necessary but reduces lock scope.

@daviddrysdale
Copy link
Contributor Author

Ah yes, I see it now, thanks for the pointer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@blaxill blaxill

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@daviddrysdale @blaxill