Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark exception stops entire suite run #1803

Closed
deipax opened this issue Sep 10, 2021 · 3 comments · Fixed by #1903
Closed

Benchmark exception stops entire suite run #1803

deipax opened this issue Sep 10, 2021 · 3 comments · Fixed by #1903
Assignees
Milestone

Comments

@deipax
Copy link

deipax commented Sep 10, 2021

Somewhere between version 0.12.1 and 0.13.1, when a Benchmark throws an exception the entire run is now aborted. Where I used to receive the message:

"No more Benchmark runs will be launched as NO measurements were obtained from the previous run!"

and the rest of the benchmarks continued to run, the entire suite now stops and reports the exception.

Edit The work around is to use 0.12.1.

@vvdev
Copy link

vvdev commented Nov 9, 2021

I experience the same issue since upgraded to 0.13.1

My problem is, that i'm trying to benchmark my code under .NET 6, and ThreadingDiagnoser from 0.12.1 doesn't work under .NET6 build.

@vvdev
Copy link

vvdev commented Nov 9, 2021

I just want to provide a little bit more context to explain why it can be an important issue:

I have benchmark suite which works about 20 hours per platform, it measures all individual aspects of solution and we run it before we commit each refactoring/change to ensure we didn't introduce any performance degradation.

Normally, after every coding day we run subset of this suite, and it then works for about 10 hours, overnight.
As per normal development process some bugs are time to time introduced and some benchmarks can fail.

And if this is the case - then you simply lost 1 day and should continue development without knowing impact of your last changes.
In situations when you literally hunting microseconds and bytes of allocations it really hurts.

PS: thank you for the great tool :)

@adamsitnik
Copy link
Member

Hi @vvdev @deipax

In the near future I won't have time to investigate the problem, but feel free to do it and send a PR with a fix. I'll definitely find some time to review the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants