serverset::tests::backoff_when_unhealthy is flaky

[stuhood]

Seen in https://travis-ci.org/pantsbuild/pants/jobs/540442640:

test tests::backoff_when_unhealthy ... thread 'tokio-runtime-worker-1' panicked at 'assertion failed: `(left == right)`
  left: `"good"`,
 right: `"bad"`', serverset/src/lib.rs:479:13
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:197
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at src/libstd/panicking.rs:474
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:381
   6: std::panicking::try::do_call
             at src/libstd/panicking.rs:336
   7: serverset::tests::expect_only_good::{{closure}}
             at serverset/src/lib.rs:479
   8: core::result::Result<T,E>::map
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/result.rs:458
   9: <futures::future::map::Map<A,F> as futures::future::Future>::poll
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/future/map.rs:35
  10: futures::future::chain::Chain<A,B,C>::poll
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/future/chain.rs:26
  11: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/future/then.rs:32
  12: <futures::task_impl::NotifyHandle as core::clone::Clone>::clone
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/future/mod.rs:113
  13: futures::task_impl::NotifyHandle::new
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/task_impl/mod.rs:329
  14: futures::task_impl::NotifyHandle::new
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/task_impl/mod.rs:399
  15: futures::task_impl::std::CURRENT_THREAD_NOTIFY::__init
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/task_impl/std/mod.rs:83
  16: futures::task_impl::NotifyHandle::new
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/task_impl/mod.rs:399
  17: futures::task_impl::NotifyHandle::new
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/task_impl/mod.rs:291
  18: futures::task_impl::NotifyHandle::new
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/futures-0.1.27/src/task_impl/mod.rs:329
  19: tokio_threadpool::task::Task::run::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/task/mod.rs:145
  20: core::ops::function::FnOnce::call_once
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/ops/function.rs:231
  21: tokio_threadpool::park::default_park::DefaultPark::park_sync
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panic.rs:309
  22: crossbeam_utils::backoff::Backoff::snooze
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panicking.rs:293
  23: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:87
  24: crossbeam_utils::backoff::Backoff::snooze
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panicking.rs:272
  25: crossbeam_utils::backoff::Backoff::snooze
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panic.rs:388
  26: tokio_threadpool::task::Task::run
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/task/mod.rs:130
  27: tokio_threadpool::worker::Worker::run_task2
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/worker/mod.rs:567
  28: tokio_threadpool::worker::Worker::run_task
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/worker/mod.rs:459
  29: tokio_threadpool::worker::Worker::try_run_owned_task
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/worker/mod.rs:390
  30: tokio_threadpool::worker::Worker::try_run_task
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/worker/mod.rs:297
  31: tokio_threadpool::worker::Worker::with_current::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/worker/mod.rs:241
  32: tokio::runtime::threadpool::builder::Builder::build::{{closure}}::{{closure}}::{{closure}}::{{closure}}::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.17/src/runtime/threadpool/builder.rs:349
  33: futures::task_impl::std::CURRENT_THREAD_NOTIFY::__init
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-trace-core-0.1.0/src/dispatcher.rs:54
  34: tokio::runtime::threadpool::builder::Builder::build::{{closure}}::{{closure}}::{{closure}}::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.17/src/runtime/threadpool/builder.rs:348
  35: <tokio_reactor::HandlePriv as core::clone::Clone>::clone
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.10/src/timer/handle.rs:94
  36: alloc::raw_vec::alloc_guard
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:299
  37: alloc::raw_vec::alloc_guard
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:245
  38: <tokio_reactor::HandlePriv as core::clone::Clone>::clone
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.10/src/timer/handle.rs:81
  39: tokio::runtime::threadpool::builder::Builder::build::{{closure}}::{{closure}}::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.17/src/runtime/threadpool/builder.rs:347
  40: futures::task_impl::core::get_ptr
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.10/src/clock/clock.rs:141
  41: alloc::raw_vec::alloc_guard
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:299
  42: alloc::raw_vec::alloc_guard
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:245
  43: futures::task_impl::core::get_ptr
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-timer-0.2.10/src/clock/clock.rs:124
  44: tokio::runtime::threadpool::builder::Builder::build::{{closure}}::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.17/src/runtime/threadpool/builder.rs:346
  45: alloc::raw_vec::alloc_guard
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.9/src/lib.rs:237
  46: alloc::raw_vec::alloc_guard
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:299
  47: alloc::raw_vec::alloc_guard
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:245
  48: alloc::raw_vec::alloc_guard
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-reactor-0.1.9/src/lib.rs:217
  49: tokio::runtime::threadpool::builder::Builder::build::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.17/src/runtime/threadpool/builder.rs:345
  50: tokio_threadpool::callback::Callback::call
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/callback.rs:22
  51: tokio_threadpool::worker::Worker::do_run::{{closure}}::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/worker/mod.rs:127
  52: tokio_threadpool::worker::CURRENT_WORKER::__init
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.6/src/global.rs:192
  53: crossbeam_epoch::collector::LocalHandle::is_pinned
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:299
  54: crossbeam_epoch::collector::LocalHandle::is_pinned
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:245
  55: tokio_threadpool::worker::CURRENT_WORKER::__init
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-executor-0.1.6/src/global.rs:162
  56: tokio_threadpool::worker::Worker::do_run::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/worker/mod.rs:125
  57: crossbeam_epoch::collector::LocalHandle::is_pinned
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:299
  58: crossbeam_epoch::collector::LocalHandle::is_pinned
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/local.rs:245
  59: tokio_threadpool::worker::Worker::do_run
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/worker/mod.rs:116
  60: tokio_threadpool::pool::Pool::spawn_thread::{{closure}}
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-threadpool-0.1.12/src/pool/mod.rs:344
  61: crossbeam_epoch::collector::LocalHandle::is_pinned
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/sys_common/backtrace.rs:136
  62: std::sync::once::Once::is_completed
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/mod.rs:470
  63: tokio_threadpool::park::default_park::DefaultPark::park_sync
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panic.rs:309
  64: crossbeam_utils::backoff::Backoff::snooze
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panicking.rs:293
  65: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:87
  66: crossbeam_utils::backoff::Backoff::snooze
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panicking.rs:272
  67: crossbeam_utils::backoff::Backoff::snooze
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panic.rs:388
  68: std::sync::once::Once::is_completed
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/thread/mod.rs:469
  69: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/ops/function.rs:231
  70: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/liballoc/boxed.rs:702
  71: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/liballoc/boxed.rs:702
             at src/libstd/sys_common/thread.rs:14
             at src/libstd/sys/unix/thread.rs:80
  72: _pthread_body
  73: _pthread_start
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Canceled', src/libcore/result.rs:997:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:59
             at src/libstd/panicking.rs:197
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:211
   4: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at src/libstd/panicking.rs:474
   5: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:381
   6: std::panicking::try::do_call
             at src/libstd/panicking.rs:308
   7: <T as core::any::Any>::type_id
             at src/libcore/panicking.rs:85
   8: core::result::unwrap_failed
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/macros.rs:18
   9: core::result::Result<T,E>::unwrap
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/result.rs:798
  10: tokio::runtime::threadpool::Runtime::block_on
             at /Users/travis/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.1.17/src/runtime/threadpool/mod.rs:266
  11: serverset::tests::expect_only_good
             at serverset/src/lib.rs:476
  12: serverset::tests::backoff_when_unhealthy
             at serverset/src/lib.rs:376
  13: serverset::tests::backoff_when_unhealthy::{{closure}}
             at serverset/src/lib.rs:358
  14: core::ops::function::FnOnce::call_once
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/ops/function.rs:231
  15: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/liballoc/boxed.rs:702
  16: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:87
  17: test::run_test::run_test_inner::{{closure}}
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panicking.rs:272
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/panic.rs:388
             at src/libtest/lib.rs:1468
  18: test::run_test::run_test_inner
             at src/libtest/lib.rs:1490
  19: test::run_tests_console::{{closure}}
             at src/libtest/lib.rs:1509
  20: test::fmt_thousands_sep
             at src/libtest/lib.rs:1167
             at src/libtest/lib.rs:972
  21: test::stats::percentile_of_sorted
             at src/libtest/lib.rs:293
  22: test::stats::percentile_of_sorted
             at src/libtest/lib.rs:327
  23: serverset::main
  24: std::rt::lang_start::{{closure}}
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/rt.rs:64
  25: std::panicking::try::do_call
             at src/libstd/rt.rs:49
             at src/libstd/panicking.rs:293
  26: panic_unwind::dwarf::eh::read_encoded_pointer
             at src/libpanic_unwind/lib.rs:87
  27: <std::panicking::begin_panic::PanicPayload<A> as core::panic::BoxMeUp>::get
             at src/libstd/panicking.rs:272
             at src/libstd/panic.rs:388
             at src/libstd/rt.rs:48
  28: std::rt::lang_start
             at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/rt.rs:64
  29: serverset::main
FAILED

[jsirois]

And again on master https://travis-ci.org/pantsbuild/pants/jobs/541548404#L537

[stuhood]

Seen again in master.

[illicitonion]

I understand what's going on here - prepping a PR :)

[jsirois]

Not dead yet. Seen in master here: https://travis-ci.org/pantsbuild/pants/jobs/588135910#L5511

...
     Running src/rust/engine/target/debug/deps/serverset-0878d4656386dc17
running 10 tests
test retry::tests::gives_up_on_enough_bad ... ok
test retry::tests::retries ... ok
test tests::backoff_when_unhealthy ... thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `"good"`,
 right: `"bad"`', serverset/src/lib.rs:606:9
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/vsts/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /Users/vsts/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:200
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:214
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:384
   8: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:339
   9: serverset::tests::expect_only_good
             at serverset/src/lib.rs:606
  10: serverset::tests::backoff_when_unhealthy
             at serverset/src/lib.rs:506
  11: serverset::tests::backoff_when_unhealthy::{{closure}}
             at serverset/src/lib.rs:484
  12: core::ops::function::FnOnce::call_once
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/ops/function.rs:231
  13: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
  14: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:82
  15: std::panicking::try
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/panicking.rs:275
  16: std::panic::catch_unwind
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/panic.rs:394
  17: test::run_test::run_test_inner::{{closure}}
             at src/libtest/lib.rs:1466
  18: test::run_test::run_test_inner
             at src/libtest/lib.rs:1488
  19: test::run_test
             at src/libtest/lib.rs:1507
  20: test::run_tests
             at src/libtest/lib.rs:1166
  21: test::run_tests_console
             at src/libtest/lib.rs:971
  22: test::test_main
             at src/libtest/lib.rs:292
  23: test::test_main_static
             at src/libtest/lib.rs:326
  24: serverset::main
  25: std::rt::lang_start::{{closure}}
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/rt.rs:64
  26: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:49
  27: std::panicking::try::do_call
             at src/libstd/panicking.rs:296
  28: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:82
  29: std::panicking::try
             at src/libstd/panicking.rs:275
  30: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  31: std::rt::lang_start_internal
             at src/libstd/rt.rs:48
  32: std::rt::lang_start
             at /rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/rt.rs:64
  33: serverset::main
FAILED
test tests::handles_overflow_internally ... ok
test tests::no_servers_is_error ... ok
test tests::one_request_works ... ok
test tests::reattempts_unhealthy ... ok
test tests::round_robins ... ok
test tests::skips_unhealthy ... ok
test tests::waits_if_all_unhealthy ... ok
failures:
failures:
    tests::backoff_when_unhealthy
test result: FAILED. 9 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
error: test failed, to rerun pass '-p serverset --lib'

[Eric-Arellano]

Seen again.

test tests::backoff_when_unhealthy ... thread 'main' panicked at 'assertion failed: `(left == right)`
485  left: `"good"`,
486 right: `"bad"`', serverset/src/tests.rs:229:7
487stack backtrace:
488   0: backtrace::backtrace::libunwind::trace
489             at /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
490   1: backtrace::backtrace::trace_unsynchronized
491             at /Users/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
492   2: std::sys_common::backtrace::_print_fmt
493             at src/libstd/sys_common/backtrace.rs:77
494   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
495             at src/libstd/sys_common/backtrace.rs:61
496   4: core::fmt::write
497             at src/libcore/fmt/mod.rs:1028
498   5: std::io::Write::write_fmt
499             at src/libstd/io/mod.rs:1412
500   6: std::sys_common::backtrace::_print
501             at src/libstd/sys_common/backtrace.rs:65
502   7: std::sys_common::backtrace::print
503             at src/libstd/sys_common/backtrace.rs:50
504   8: std::panicking::default_hook::{{closure}}
505             at src/libstd/panicking.rs:188
506   9: std::panicking::default_hook
507             at src/libstd/panicking.rs:205
508  10: std::panicking::rust_panic_with_hook
509             at src/libstd/panicking.rs:464
510  11: std::panicking::continue_panic_fmt
511             at src/libstd/panicking.rs:373
512  12: std::panicking::begin_panic_fmt
513             at src/libstd/panicking.rs:328
514  13: serverset::tests::expect_only_good
515             at serverset/src/tests.rs:229
516  14: serverset::tests::backoff_when_unhealthy
517             at serverset/src/tests.rs:125
518  15: serverset::tests::backoff_when_unhealthy::{{closure}}
519             at serverset/src/tests.rs:107
520  16: core::ops::function::FnOnce::call_once
521             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/ops/function.rs:227
522  17: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
523             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/liballoc/boxed.rs:942
524  18: __rust_maybe_catch_panic
525             at src/libpanic_unwind/lib.rs:78
526  19: std::panicking::try
527             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panicking.rs:265
528  20: std::panic::catch_unwind
529             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/panic.rs:396
530  21: test::run_test_in_process
531             at src/libtest/lib.rs:570
532  22: test::run_test::run_test_inner::{{closure}}
533             at src/libtest/lib.rs:473
534  23: test::run_test::run_test_inner
535             at src/libtest/lib.rs:494
536  24: test::run_test
537             at src/libtest/lib.rs:530
538  25: test::run_tests
539             at src/libtest/lib.rs:299
540  26: test::console::run_tests_console
541             at src/libtest/console.rs:295
542  27: test::test_main
543             at src/libtest/lib.rs:120
544  28: test::test_main_static
545             at src/libtest/lib.rs:139
546  29: serverset::main
547  30: std::rt::lang_start::{{closure}}
548             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/rt.rs:61
549  31: std::rt::lang_start_internal::{{closure}}
550             at src/libstd/rt.rs:48
551  32: std::panicking::try::do_call
552             at src/libstd/panicking.rs:287
553  33: __rust_maybe_catch_panic
554             at src/libpanic_unwind/lib.rs:78
555  34: std::panicking::try
556             at src/libstd/panicking.rs:265
557  35: std::panic::catch_unwind
558             at src/libstd/panic.rs:396
559  36: std::rt::lang_start_internal
560             at src/libstd/rt.rs:47
561  37: std::rt::lang_start
562             at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/rt.rs:61
563  38: serverset::main
564note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
565FAILED

@illicitonion @gshuflin there's no way to mark a test as flaky in Rust / with Cargo, correct? Per our flaky test policy, the first step is to mark as flaky and give it 1 retry, then if it continues to flake put up a PR to delete it. If we care enough about preserving the test, we can add it back in a followup PR that fixes the original flakiness. If we can't mark tests as flaky, then we jump straight to removing the test.

[stuhood]

No longer exists.