-
-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for running criterion benches as tests? #96
Comments
Thanks for reporting this! I think the easiest thing to do would be to make criterion support the --format terse option and output lines in the format nextest accepts: https://nexte.st/book/custom-test-harnesses.html Note that following the standard Rust benchmark library, lines ending in ": bench" are ignored. We may want to add an option to run benchmarks as tests if that's a desired use case (is that what you want?) |
Thanks for the response! That makes sense, I missed that section in the book. I'll open an issue over on criterion to see if they'd be open to adding those flags.
Yeah, we have had cases where benches were broken accidentally so we started running them as tests in CI which seems to just run one benchmark iteration and lets us validate that no assertions failed (for criterion at least). That would be great if |
`nextest` doesn't work with criterion benches: nextest-rs/nextest#96 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
`nextest` doesn't work with criterion benches: nextest-rs/nextest#96 Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
There's also #38 which might be useful. |
* chore: Try out nextest in CI Starting with flakey test retries. Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Split up benches and non-benches tests `nextest` doesn't work with criterion benches: nextest-rs/nextest#96 Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Avoid testing benches with nextest Since criterion doesn't support the flags mentioned on https://nexte.st/book/custom-test-harnesses.html Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Fix cargo nextest installation Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Run docs tests too since nextest doesn't support them Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Fix docs test Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Don't test benches since there isn't a way to test just them Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Install nextest for Windows Signed-off-by: Jesse Szwedko <jesse@szwedko.me> * Fix CLI test running Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
I think we can support this with a |
I looked at this a bit more and it seems like criterion binaries built with cargo test don't appear to behave with
I think this will need to be fixed in criterion first -- marking this "help wanted" in case someone wants to pick up the work in criterion. See https://nexte.st/book/custom-test-harnesses for more documentation (note that we also accept lines ending with |
Thanks for following up @sunshowers ! I agree bheisler/criterion.rs#562 will need to be resolved first. |
bheisler/criterion.rs#602 addresses this. |
Criterion 0.5, which came out a few days ago, supports nextest. |
Hi all!
I'm a big fan of what this project is doing.
I noticed when trying to integrate this into https://github.com/vectordotdev/vector that it fails to run test binaries built from criterion benchmarks which don't support the same
--format
flag that normal test binaries support:Running
--help
on the test binary:I was just curious to get thoughts on handling this. Should I stick with normal
cargo test --benches
for that target for now and usenextest
for the other targets?The text was updated successfully, but these errors were encountered: