--rerun-fails reruns all tests in packages if packages are also passed to go test args

[matthewhughes-uw]

That is, if you run gotestsum --rerun-fails --packages ./... -- ../.. then on failure all packages matching ./... will be re-tested. I don't think this is necessarily a bug, but might warrant some more explicit documentation

Reproduction:

$ go mod init module
$ mkdir --parents pkg/foo pkg/bar

// pkg/foo/foo.go
package foo

func BoringFunc() int {
	return 3
}

// pkg/foo/foo_test.go
package foo

import (
	"testing"
)

func TestBoringFunc(t *testing.T) {
	if BoringFunc() != 3 {
		t.Fatal("wrong")
	}
}

// pkg/bar/bar.go
package bar

func BoringFunc() int {
	return 4
}

// pkg/bar/bar_test.go
package bar

import (
	"testing"
)

func TestBoringFunc(t *testing.T) {
	if BoringFunc() != 3 {
		t.Fatal("wrong")
	}
}

running gotestsum --rerun-fails --packages ./... re-runs only the failing test, as expected:

$ gotestsum --rerun-fails --packages ./... -- -count=1
✓  pkg/foo (2ms)
✖  pkg/bar (2ms)

DONE 2 tests, 1 failure in 0.188s

✖  pkg/bar (2ms)

DONE 2 runs, 3 tests, 2 failures in 0.371s

✖  pkg/bar (2ms)

=== Failed
=== FAIL: pkg/bar TestBoringFunc (0.00s)
    race_test.go:9: wrong

=== FAIL: pkg/bar TestBoringFunc (re-run 1) (0.00s)
    race_test.go:9: wrong

=== FAIL: pkg/bar TestBoringFunc (re-run 2) (0.00s)
    race_test.go:9: wrong

DONE 3 runs, 4 tests, 3 failures in 0.539s

However, when run with a list of packages passed as args, all tests are re-run:

$ gotestsum --rerun-fails --packages ./... -- -count=1 ./...
✖  pkg/bar (2ms)
✓  pkg/foo (2ms)

DONE 2 tests, 1 failure in 0.209s

✖  pkg/bar (2ms)
✓  pkg/foo (2ms)

DONE 2 runs, 4 tests, 2 failures in 0.395s

✖  pkg/bar (2ms)
✓  pkg/foo (2ms)

=== Failed
=== FAIL: pkg/bar TestBoringFunc (0.00s)
    race_test.go:9: wrong

=== FAIL: pkg/bar TestBoringFunc (re-run 1) (0.00s)
    race_test.go:9: wrong

=== FAIL: pkg/bar TestBoringFunc (re-run 2) (0.00s)
    race_test.go:9: wrong

DONE 3 runs, 6 tests, 3 failures in 0.556s

Maybe it's worth updating the docs to mention how packages should (or shouldn't) be passed when using --rerun-fails?

Further, I guess gotestsum could try and determine if the user passed both --packages and a list of packages as args to go test and do something then? But I think an update to the docs would be enough.

I ran into this behaviour in a private repo at my work, but here's another example I found in the wild: https://github.com/hashicorp/nomad/blob/54aafa574d2dc4bd281b28f45dc00593c1e337da/GNUmakefile#L287

[dnephin]

Thank you for the bug report! I wonder if some are using this as a way to re-run everything on any failure. I do something similar occasionally by specifying extra packages to gotestsum --watch.

I think a warning log message and a docs update would be great!

[matthewhughes934]

I think a warning log message and a docs update would be great!

Looking into this, is there an existing solution to seeing if a list of packages was passed to go test? For example:

• gotestsum --rerun-fails --packages ./... -- ./... should emit a warning
• gotestsum --rerun-fails --packages ./... -- -coverprofile 'some.file' shouldn't emit a warning