Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Run tests faster with nextest #778

Merged
merged 5 commits into from
Jun 22, 2022
Merged

Conversation

Hywan
Copy link
Member

@Hywan Hywan commented Jun 20, 2022

cargo-nextest is a next-generation
test runner for Rust projects.

This patch installs and uses nextest to run our own tests.

Comparing cargo test and cargo nextest with hyperfine provides the
following results:

$ hyperfine 'cargo test --workspace' 'cargo nextest run --workspace && cargo test --doc'
Benchmark 1: cargo test --workspace
  Time (mean ± σ):     51.785 s ±  2.066 s    [User: 183.471 s, System: 10.563 s]
  Range (min … max):   49.151 s … 56.641 s    10 runs

Benchmark 2: cargo nextest run --workspace && cargo test --doc
  Time (mean ± σ):     44.556 s ±  0.894 s    [User: 192.213 s, System: 11.441 s]
  Range (min … max):   43.170 s … 45.762 s    10 runs

Benchmark 2 is 1.16 times faster than Benchmark 1

Real numbers are described in #778 (comment).

@Hywan Hywan added the enhancement New feature or request label Jun 20, 2022
@jplatte
Copy link
Collaborator

jplatte commented Jun 20, 2022

nextest doesn't support doctests. We will have to run cargo test --doc as well (and add it to nextest's time in the comparison).

@codecov
Copy link

codecov bot commented Jun 20, 2022

Codecov Report

Merging #778 (fdca461) into main (87639a4) will increase coverage by 0.02%.
The diff coverage is n/a.

❗ Current head fdca461 differs from pull request most recent head b0d51fd. Consider uploading reports for the commit b0d51fd to get more accurate results

@@            Coverage Diff             @@
##             main     #778      +/-   ##
==========================================
+ Coverage   77.37%   77.39%   +0.02%     
==========================================
  Files          94       94              
  Lines       13735    13730       -5     
==========================================
- Hits        10627    10626       -1     
+ Misses       3108     3104       -4     
Impacted Files Coverage Δ
crates/matrix-sdk-crypto/src/store/mod.rs 53.33% <0.00%> (-1.63%) ⬇️
crates/matrix-sdk-base/src/store/ambiguity_map.rs 82.35% <0.00%> (ø)
...tes/matrix-sdk-base/src/store/integration_tests.rs 97.46% <0.00%> (ø)
crates/matrix-sdk-base/src/store/mod.rs 81.70% <0.00%> (+1.70%) ⬆️
crates/matrix-sdk/src/client/builder.rs 71.11% <0.00%> (+4.44%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87639a4...b0d51fd. Read the comment docs.

Copy link
Collaborator

@jplatte jplatte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the figures from the first commit description up-to-date now?

xtask/src/ci.rs Show resolved Hide resolved
@jplatte jplatte mentioned this pull request Jun 20, 2022
@Hywan
Copy link
Member Author

Hywan commented Jun 20, 2022

We may be able to run cargo nextest in parallel of cargo test --doc with GNU parallel, or https://gitlab.redox-os.org/redox-os/parallel for having something cross-platform, but we should have prebuilt binaries, or cache, otherwise the compilation-time with cost us too much. Let's see with the current numbers.

Edit: xargs -P can also be useful in this case.

@Hywan
Copy link
Member Author

Hywan commented Jun 20, 2022

With xargs -P, we can achieve the following results:

Benchmark 1: cargo test --workspace
  Time (mean ± σ):     51.785 s ±  2.066 s    [User: 183.471 s, System: 10.563 s]
  Range (min … max):   49.151 s … 56.641 s    10 runs

Benchmark 2: cargo nextest run --workspace && cargo test --doc
  Time (mean ± σ):     44.556 s ±  0.894 s    [User: 192.213 s, System: 11.441 s]
  Range (min … max):   43.170 s … 45.762 s    10 runs

Benchmark 3: echo "cargo nextest run --workspace\0cargo test --doc" | xargs -0 -P 0 -I@ sh -c @
  Time (mean ± σ):     34.339 s ±  4.357 s    [User: 192.472 s, System: 11.739 s]
  Range (min … max):   22.009 s … 36.422 s    10 runs

Benchmark 2 is 1.16 times faster than Benchmark 1
Benchmark 3 is 1.51 times faster than Benchmark 1

I'm not sure how it is possible to run such xargs command in a YAML file while having cross-compatibility, but let's try!

@Hywan
Copy link
Member Author

Hywan commented Jun 20, 2022

Without xargs -P, here are the results for the “test” step of the CI workflow for some jobs (I won't do it exhaustively, time differs, it's just to provide an order of magnitude idea):

jobs before  after speed up factor
linux / stable 11m 21s 9m 13s 1.23
macOS / stable 15m 25s 10m 40s 1.45
linux / features-sso-login 9m 30s 5m 9s 1.84
linux / features-rustls-tls 5m 4s 2m 13s 2.29
linux / features-no-encryption 6m 16s 2m 14s 2.81
linux / features-no-sled 6m 51s 4m 9s 1.65
linux / features-no-encryption-and-sled 5m 10s 1m 51s 2.79

Just on those examples, running tests is 2.01 times faster on average.

@Hywan
Copy link
Member Author

Hywan commented Jun 20, 2022

When running cargo nextest run and cargo test --doc in parallel, we have worst results on Github Actions, probably because the number of threads is limited. In that case, they are spending more time waiting on each other than truly running what we want. Let's keep things simple and let's use cargo-nextest for now.

Copy link
Collaborator

@jplatte jplatte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removal of dead code is still not a separate commit afaict but otherwise looks good.

> [`cargo-nextest`](https://nexte.st/index.html) is a next-generation
> test runner for Rust projects.

This patch installs and uses `nextest` to run our own tests.

Comparing `cargo test` and `cargo nextest` with hyperfine provides the
following results:

```sh
$ hyperfine 'cargo test --workspace' 'cargo nextest run --workspace && cargo test --doc'
Benchmark 1: cargo test --workspace
  Time (mean ± σ):     51.785 s ±  2.066 s    [User: 183.471 s, System: 10.563 s]
  Range (min … max):   49.151 s … 56.641 s    10 runs

Benchmark 2: cargo nextest run --workspace && cargo test --doc
  Time (mean ± σ):     44.556 s ±  0.894 s    [User: 192.213 s, System: 11.441 s]
  Range (min … max):   43.170 s … 45.762 s    10 runs
```

Benchmark 2 is 1.16 times faster than Benchmark 1.
`cargo-nextest` doesn't support doctests for now, so we must run them
“manually” by running a separate `cargo test --doc` command.
This patch also changes the step's name from Clippy to Test.
@Hywan
Copy link
Member Author

Hywan commented Jun 22, 2022

Removal of dead code is still not a separate commit afaict but otherwise looks good.

Done, d9475c1 :-).

@Hywan Hywan enabled auto-merge June 22, 2022 07:28
@Hywan Hywan merged commit 6ad323b into matrix-org:main Jun 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants