-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Successful benchmark run is marked as failed #332
Comments
Thank you for the bug report! I ran this example with
The problem in #62 does still appear to present, but I think this is a new regression in go1.20 (#322 is a similar problem). The
I'm not sure what to do about this. I haven't yet searched the Go issue tracker to see if someone else has reported the problem. |
This is golang/go#61767 |
gotestsum can't properly process benchmark results due to a go toolchain bug. Remove the postprocessing, since we don't benefit as much now that we don't use Semaphore CI anymore (which had nice visualisation for JUnit output). See gotestyourself/gotestsum#332 Signed-off-by: Lorenz Bauer <lmb@isovalent.com>
Execute benchmarks once on CI, to prevent bitrot from setting in. We do this as a separate target without gotestsum filtering since there is a bug in the Go toolchain which prevents benchmark output from being parsed properly. See gotestyourself/gotestsum#332 Signed-off-by: Lorenz Bauer <lmb@isovalent.com>
Execute benchmarks once on CI, to prevent bitrot from setting in. We do this as a separate target without gotestsum filtering since there is a bug in the Go toolchain which prevents benchmark output from being parsed properly. See gotestyourself/gotestsum#332 Signed-off-by: Lorenz Bauer <lmb@isovalent.com>
The value added is quite low and there is currently a bug marking the tests as failed, which is confusing. See gotestyourself/gotestsum#332.
Hey all,
we observed gotestsum fail our benchmarks despite the benchmarks running fine. I tried to reproduce the issue and this was the smallest setup I could reproduce the issue with (using the latest gotestsum release v1.10.0):
Given a simple benchmark
when run as e.g.
it should not mark the test as failed.
go version
isgo version go1.20.4 darwin/arm64
.I found #62, is this related? Or do we use some incompatible configuration?
The text was updated successfully, but these errors were encountered: