Availability recovery bench hangs test runner

[ggwpez]

Trying to run our tests now does not even start anymore with creating test list failed:

$ cargo nextest run --workspace --no-fail-fast

Finished test [unoptimized + debuginfo] target(s) in 4m 59s
error: creating test list failed

Caused by:
  for `polkadot-availability-recovery::availability-recovery-regression-bench`, command `/home/oliver/remote-builds/2722749496389579917/target/debug/deps/availability_recovery_regression_bench-b1fdf52926626227 --list --format terse` exited with code 1
--- stdout:
Warming up...
Took 0 runs to warm up

data_availability_read

Network usage, KiB                     total   per block
Sent to peers                          5.000       1.667
Received from peers               921609.000  307203.000

CPU usage, seconds                     total   per block
Test environment                       1.007       0.336
availability-recovery                 50.853      16.951


--- stderr:
data_availability_read: The resource `availability-recovery` is expected to be equal to 11.5 with a precision 0.05, but the current value is 16.950935916999995
Error: "Regressions found"

[AndreiEres]

@ggwpez I'm very sorry for that. cargo test doesn't run it because these test require subsystem-benchmarks feature. Unfortunately, nextext ignores required features, so in CI we add this --filter-expr 'not deps(/polkadot-subsystem-bench/)'.

[ggwpez]

Oh, so it is feature gated, okay.
I think nextest tries to call --list on the bench since it is marked as test, and that fails.
Maybe you can make it a no-op if the feature is not enabled?

[bkchr]

Unfortunately, nextext ignores required features, so in CI we add this --filter-expr 'not deps(/polkadot-subsystem-bench/)'.

I don't think that this is the issue. The issue is that you depend on polkadot-subsystem-bench in your dev dependencies and that depends on availability-recovery with the subsystem-benchmarks feature. Which basically means that the feature is always enabled when you run cargo test.

[AndreiEres]

The issue is that you depend on polkadot-subsystem-bench in your dev dependencies and that depends on availability-recovery with the subsystem-benchmarks feature.

Yes, Oliver already found it and I prepared a fix here #3716