Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remote::tests::retry_for_cancelled_channel is flaky #7149

Closed
stuhood opened this issue Jan 24, 2019 · 2 comments
Closed

remote::tests::retry_for_cancelled_channel is flaky #7149

stuhood opened this issue Jan 24, 2019 · 2 comments
Labels
flaky-test

Comments

@stuhood
Copy link
Sponsor Member

stuhood commented Jan 24, 2019 

test remote::tests::retry_for_cancelled_channel ... thread 'remote::tests::retry_for_cancelled_channel' panicked at 'called `Result::unwrap()` on an `Err` value: "Error getting execution_client: Error connecting to remote execution server: Error attempting to establish underlying session layer: Operation timed out (os error 60)"', src/libcore/result.rs:1009:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:211
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:227
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at src/libstd/panicking.rs:491
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:398
   6: std::panicking::try::do_call
             at src/libstd/panicking.rs:325
   7: core::char::methods::<impl char>::escape_debug
             at src/libcore/panicking.rs:95
   8: <core::str::Chars<'a> as core::iter::iterator::Iterator>::next
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/macros.rs:26
   9: <core::result::Result<T, E>>::unwrap
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/result.rs:808
  10: process_execution::remote::tests::retry_for_cancelled_channel
             at process_execution/src/remote.rs:1612
  11: process_execution::remote::tests::retry_for_cancelled_channel::{{closure}}
             at process_execution/src/remote.rs:1588
  12: core::ops::function::FnOnce::call_once
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
  13: <F as alloc::boxed::FnBox<A>>::call_box
             at src/libtest/lib.rs:1471
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:673
  14: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:102
thread 'remote::tests::retry_for_cancelled_channel' panicked at 'assertion failed: `(left == right)`
  left: `3`,
 right: `0`: Expected 3 more requests. Remaining expected responses:
MockOperation { op: Ok(Some(Operation { name: "gimme-foo", metadata: None, done: false, result: None })), duration: None }
MockOperation { op: Ok(None), duration: Some(100ms) }
MockOperation { op: Ok(Some(Operation { name: "gimme-foo", metadata: None, done: true, result: Some(Response(Any { type_url: "type.googleapis.com/build.bazel.remote.execution.v2.ExecuteResponse", value: [10, 5, 42, 3, 102, 111, 111] })) })), duration: None }
Received requests:
', testutil/mock/src/execution_server.rs:127:5
stack backtrace:
   0:        0x1077e10d3 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h16b82ef029078385
                               at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1:        0x1077db66c - std::sys_common::backtrace::_print::hc39c8623f5320a3a
                               at src/libstd/sys_common/backtrace.rs:71
   2:        0x1077dfb91 - std::panicking::default_hook::{{closure}}::h928a543b2f8eb1f2
                               at src/libstd/sys_common/backtrace.rs:59
                               at src/libstd/panicking.rs:211
   3:        0x1077df88d - std::panicking::default_hook::hb3ba6facbede384c
                               at src/libstd/panicking.rs:227
   4:        0x1077e031e - <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get::h6db94dbda67cef59
                               at src/libstd/panicking.rs:491
   5:        0x1077dfe3c - std::panicking::continue_panic_fmt::hf7169aba6b1afe9c
                               at src/libstd/panicking.rs:398
   6:        0x1077dfd90 - std::panicking::try::do_call::hb3270f6ae846849d
                               at src/libstd/panicking.rs:353
   7:        0x106cc0d95 - <bytes::bytes::BytesMut as bytes::buf::buf_mut::BufMut>::put_slice::h0ffee61823f4ab22
                               at testutil/mock/src/execution_server.rs:127
   8:        0x106a656c4 - core::ptr::real_drop_in_place::h5250ec6b9c21b554
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ptr.rs:204
   9:        0x106c073b7 - process_execution::remote::tests::retry_for_cancelled_channel::hf59a281b0c350d27
                               at process_execution/src/remote.rs:1624
  10:        0x106a8d970 - process_execution::remote::tests::retry_for_cancelled_channel::{{closure}}::h2fbdd16c747b9a13
                               at process_execution/src/remote.rs:1588
  11:        0x106a5b680 - core::ops::function::FnOnce::call_once::he4e4465b2eb836f6
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
  12:        0x106c33621 - <F as alloc::boxed::FnBox<A>>::call_box::h0b961cc85c049bee
                               at src/libtest/lib.rs:1471
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:673
  13:        0x1077f3d6e - panic_unwind::dwarf::eh::read_encoded_pointer::hd3e9a1f2bee55ce8
                               at src/libpanic_unwind/lib.rs:102
  14:        0x106c2b7e5 - std::sys_common::backtrace::__rust_begin_short_backtrace::h07f538384f587585
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:289
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panic.rs:398
                               at src/libtest/lib.rs:1426
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/sys_common/backtrace.rs:136
  15:        0x106c2c5d4 - std::panicking::try::do_call::h4953be8a0738d6ec
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/thread/mod.rs:477
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panic.rs:319
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:310
  16:        0x1077f3d6e - panic_unwind::dwarf::eh::read_encoded_pointer::hd3e9a1f2bee55ce8
                               at src/libpanic_unwind/lib.rs:102
  17:        0x106c336f4 - <F as alloc::boxed::FnBox<A>>::call_box::h5a5d3b35dc8857f3
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:289
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panic.rs:398
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/thread/mod.rs:476
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:673
  18:        0x1077e7f7b - std::sys::unix::thread::Thread::new::thread_start::hb66fd5e16b37cfd7
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:683
                               at src/libstd/sys_common/thread.rs:24
                               at src/libstd/sys/unix/thread.rs:90
  19:     0x7fffab32593a - _pthread_body
  20:     0x7fffab325886 - _pthread_start
thread panicked while panicking. aborting.
@cosmicexplorer
Copy link
Contributor

I believe remote::tests::wait_between_request_3_retry is similarly flaky: see https://travis-ci.org/pantsbuild/pants/jobs/484164323:

value: "Error { kind: Inner(Error { kind: Proto(PROTOCOL_ERROR) }) }"', src/libcore/result.rs:1009:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:211
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:227
   4: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:491
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:398
   6: rust_begin_unwind
             at src/libstd/panicking.rs:325
   7: core::panicking::panic_fmt
             at src/libcore/panicking.rs:95
   8: core::result::unwrap_failed
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/macros.rs:26
   9: <core::result::Result<T, E>>::unwrap
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/result.rs:808
  10: process_execution::remote::tests::wait_between_request_3_retry
             at process_execution/src/remote.rs:2301
  11: process_execution::remote::tests::wait_between_request_3_retry::{{closure}}
             at process_execution/src/remote.rs:2277
  12: core::ops::function::FnOnce::call_once
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
  13: <F as alloc::boxed::FnBox<A>>::call_box
             at src/libtest/lib.rs:1471
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
             at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:673
  14: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:102
thread 'remote::tests::wait_between_request_3_retry' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`: Expected 1 more requests. Remaining expected responses:
MockOperation { op: Ok(Some(Operation { name: "gimme-foo", metadata: None, done: true, result: Some(Response(Any { type_url: "type.googleapis.com/build.bazel.remote.execution.v2.ExecuteResponse", value: [10, 5, 42, 3, 102, 111, 111] })) })), duration: None }
Received requests:
ReceivedMessage { message_type: "ExecuteRequest", message: action_digest {hash: "a0777a41a9341ca91def500e9723de783a1c6b212d9ce05b451dd6086c0bb421" size_bytes: 138}, received_at: Instant { tv_sec: 541, tv_nsec: 617891338 } }
ReceivedMessage { message_type: "GetOperationRequest", message: name: "gimme-foo", received_at: Instant { tv_sec: 542, tv_nsec: 119798977 } }
ReceivedMessage { message_type: "GetOperationRequest", message: name: "gimme-foo", received_at: Instant { tv_sec: 543, tv_nsec: 121854790 } }
', testutil/mock/src/execution_server.rs:127:5
stack backtrace:
   0:     0x55b2b7446563 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h00d1e05a61bd440b
                               at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1:     0x55b2b7440678 - std::sys_common::backtrace::_print::hc0d53aca8da62f75
                               at src/libstd/sys_common/backtrace.rs:71
   2:     0x55b2b7444f42 - std::panicking::default_hook::{{closure}}::h46d30bcc4bfff149
                               at src/libstd/sys_common/backtrace.rs:59
                               at src/libstd/panicking.rs:211
   3:     0x55b2b7444cad - std::panicking::default_hook::h017696c2a8b7b16f
                               at src/libstd/panicking.rs:227
   4:     0x55b2b7445650 - std::panicking::rust_panic_with_hook::h8cbdfe43764887be
                               at src/libstd/panicking.rs:491
   5:     0x55b2b74451d1 - std::panicking::continue_panic_fmt::h3d3c5a833c00a5e1
                               at src/libstd/panicking.rs:398
   6:     0x55b2b744511e - std::panicking::begin_panic_fmt::h11fdc4cc73917110
                               at src/libstd/panicking.rs:353
   7:     0x55b2b681e3f8 - <mock::execution_server::TestServer as core::ops::drop::Drop>::drop::h941b01c906557114
                               at testutil/mock/src/execution_server.rs:127
   8:     0x55b2b65b9874 - core::ptr::real_drop_in_place::hcc6e81d16ea62697
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ptr.rs:204
   9:     0x55b2b6788fc3 - process_execution::remote::tests::wait_between_request_3_retry::h80a6fd8cba665f3f
                               at process_execution/src/remote.rs:2329
  10:     0x55b2b65e9e29 - process_execution::remote::tests::wait_between_request_3_retry::{{closure}}::hd3d189e447fcd371
                               at process_execution/src/remote.rs:2277
  11:     0x55b2b65a664d - core::ops::function::FnOnce::call_once::hefae193712449484
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
  12:     0x55b2b67b509e - <F as alloc::boxed::FnBox<A>>::call_box::h8c5d3aa9527f6ec6
                               at src/libtest/lib.rs:1471
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libcore/ops/function.rs:238
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:673
  13:     0x55b2b745a0d9 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:102
  14:     0x55b2b67aced3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h65c195ad031fb1f0
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:289
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panic.rs:398
                               at src/libtest/lib.rs:1426
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/sys_common/backtrace.rs:136
  15:     0x55b2b67add84 - std::panicking::try::do_call::h066a6115982fd31a
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/thread/mod.rs:477
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panic.rs:319
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:310
  16:     0x55b2b745a0d9 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:102
  17:     0x55b2b67b4eac - <F as alloc::boxed::FnBox<A>>::call_box::h4d8606d9fba2ffef
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panicking.rs:289
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/panic.rs:398
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/libstd/thread/mod.rs:476
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:673
  18:     0x55b2b744dc8d - std::sys::unix::thread::Thread::new::thread_start::hca8e72c41fa9d291
                               at /rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src/liballoc/boxed.rs:683
                               at src/libstd/sys_common/thread.rs:24
                               at src/libstd/sys/unix/thread.rs:90
  19:     0x7f7e981a36b9 - start_thread
  20:     0x7f7e97cc341c - clone
  21:                0x0 - <unknown>
thread panicked while panicking. aborting.
error: process didn't exit successfully: `/home/travis/build/pantsbuild/pants/src/rust/engine/target/debug/deps/process_execution-1895f6ba70aa0861  --nocapture` (signal: 4, SIGILL: illegal instruction)

I also ran into thread panicked while panicking. aborting. separately when working on #7116 and I believe this particular message (separate from the error itself) may be due to an error in our panic handler that causes it to panic while creating a message, but I didn't write it down and am hazy on the details.

@stuhood stuhood mentioned this issue Jan 30, 2019
Merged
@jsirois jsirois self-assigned this Mar 24, 2020
jsirois added a commit to jsirois/pants that referenced this issue Mar 25, 2020
@jsirois
Skip all flaky tests with GitHub issues.
ec40520 
This brings our skip count up from 7 python and 2 rust to 25 python and
4 rust, hopefully in the name of stability as outlined in pantsbuild#9386.

Also updates the How to Contribute docs with a pointer to pantsbuild#9386 for
the latest procedure.

Includes flaky tests from:
 pantsbuild#9313 pantsbuild#9312 pantsbuild#8679 pantsbuild#8678 pantsbuild#8520 pantsbuild#8520 pantsbuild#8520 pantsbuild#8405 pantsbuild#8193 pantsbuild#8171 pantsbuild#7836 pantsbuild#7622
 pantsbuild#7573 pantsbuild#7572 pantsbuild#7554 pantsbuild#7547 pantsbuild#7457 pantsbuild#7168 pantsbuild#7149 pantsbuild#6838 pantsbuild#6829 pantsbuild#6787 pantsbuild#6114

[ci skip-jvm-tests]  # No JVM changes made.
@jsirois jsirois mentioned this issue Mar 25, 2020
Merged
jsirois added a commit that referenced this issue Mar 25, 2020
@jsirois
Skip all flaky tests with GitHub issues. (#9391)
f393afd 
This brings our skip count up from 7 python and 2 rust to 25 python and
4 rust, hopefully in the name of stability as outlined in #9386.

Also updates the How to Contribute docs with a pointer to #9386 for
the latest procedure.

Includes flaky tests from:
 #9313 #9312 #8679 #8678 #8520 #8520 #8520 #8405 #8193 #8171 #7836 #7622
 #7573 #7572 #7554 #7547 #7457 #7168 #7149 #6838 #6829 #6787 #6114
@jsirois jsirois removed their assignment Mar 25, 2020
@benjyw
Copy link
Sponsor Contributor

benjyw commented Sep 22, 2023

These tests no longer exist, and the only ignored Rust test remaining in the repo is unrelated. Closing.

@benjyw benjyw closed this as completed Sep 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky-test
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@stuhood @jsirois @benjyw @cosmicexplorer