remote::tests::dropped_request_cancels is flaky

[illicitonion]

thread 'remote::tests::dropped_request_cancels' panicked at 'assertion failed: `(left == right)`
  left: `["gimme-foo"]`,
 right: `[]`', process_execution/src/remote.rs:3702:5
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.34/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:200
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:214
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:477
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:384
   8: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:339
   9: process_execution::remote::tests::assert_cancellation_requests
             at process_execution/src/remote.rs:3702
  10: process_execution::remote::tests::dropped_request_cancels
             at process_execution/src/remote.rs:2250
  11: process_execution::remote::tests::dropped_request_cancels::{{closure}}
             at process_execution/src/remote.rs:2173
  12: core::ops::function::FnOnce::call_once
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libcore/ops/function.rs:235
  13: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/liballoc/boxed.rs:787
  14: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:80
  15: std::panicking::try
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libstd/panicking.rs:275
  16: std::panic::catch_unwind
             at /rustc/625451e376bb2e5283fc4741caa0a3e8a2ca4d54/src/libstd/panic.rs:394
  17: test::run_test::run_test_inner::{{closure}}
             at src/libtest/lib.rs:1408
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Seen on master at https://travis-ci.org/pantsbuild/pants/jobs/594614141

[wisechengyi]

seen again at https://travis-ci.org/pantsbuild/pants/jobs/600196066

[stuhood]

Seen again in master.

[jsirois]

Seen on PR #8615.

[Eric-Arellano]

Seen again. This occurs very frequently.

hread 'remote_tests::dropped_request_cancels' panicked at 'called `Result::unwrap()` on an `Err` value: "InvalidArgument: \"Did not expect further requests from client.\""', src/libcore/result.rs:1165:5
874stack backtrace:
875   0: backtrace::backtrace::libunwind::trace
876             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/libunwind.rs:88
877   1: backtrace::backtrace::trace_unsynchronized
878             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.37/src/backtrace/mod.rs:66
879   2: std::sys_common::backtrace::_print_fmt
880             at src/libstd/sys_common/backtrace.rs:76
881   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
882             at src/libstd/sys_common/backtrace.rs:60
883   4: core::fmt::write
884             at src/libcore/fmt/mod.rs:1030
885   5: std::io::Write::write_fmt
886             at src/libstd/io/mod.rs:1412
887   6: std::sys_common::backtrace::_print
888             at src/libstd/sys_common/backtrace.rs:64
889   7: std::sys_common::backtrace::print
890             at src/libstd/sys_common/backtrace.rs:49
891   8: std::panicking::default_hook::{{closure}}
892             at src/libstd/panicking.rs:196
893   9: std::panicking::default_hook
894             at src/libstd/panicking.rs:210
895  10: std::panicking::rust_panic_with_hook
896             at src/libstd/panicking.rs:473
897  11: std::panicking::continue_panic_fmt
898             at src/libstd/panicking.rs:380
899  12: rust_begin_unwind
900             at src/libstd/panicking.rs:307
901  13: core::panicking::panic_fmt
902             at src/libcore/panicking.rs:85
903  14: core::result::unwrap_failed
904             at src/libcore/result.rs:1165
905  15: core::result::Result<T,E>::unwrap
906             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libcore/result.rs:933
907  16: process_execution::remote_tests::dropped_request_cancels
908             at process_execution/src/remote_tests.rs:1149
909  17: process_execution::remote_tests::dropped_request_cancels::{{closure}}
910             at process_execution/src/remote_tests.rs:1084
911  18: core::ops::function::FnOnce::call_once
912             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libcore/ops/function.rs:227
913  19: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
914             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/liballoc/boxed.rs:922
915  20: __rust_maybe_catch_panic
916             at src/libpanic_unwind/lib.rs:80
917  21: std::panicking::try
918             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panicking.rs:271
919  22: std::panic::catch_unwind
920             at /rustc/4560ea788cb760f0a34127156c78e2552949f734/src/libstd/panic.rs:394
921  23: test::run_test::run_test_inner::{{closure}}
922             at src/libtest/lib.rs:1413

[benjyw]

This test is no longer skipped, and seems to pass consistently.