testing: reconsider adding Context method to testing.T

[rogpeppe]

There have been various previous proposals (#16221 #18182 #18199 #18368) to add a Context method to testing.T. #16221 was even accepted and implemented but then reverted.

By my understanding, the principal argument against adding a Context method was that Context provides a way to tell things to shut down, but no way to wait for things to actually finish, so adding this functionality doesn't actually provide a good way to wait for tests to gracefully shut down, because they might fail and resources might continue to be used even after the testing package has marked the tests as successful.

For example @bcmills wrote:

A Context that is canceled when the test fails would be fine, but trying to use it in isolation to signal the return of the Test function seems like a mistake. Context cancellation is asynchronous, so nothing guarantees that the goroutines started as part of the test have actually returned when the test ends, which means they are likely to interfere with subsequent tests.

Similarly from @niemeyer here:

Such a flagging mechanism that by definition will only be useful when things will continue running in the background past the test completion does feel like a wart to me. We should be discouraging people from doing that in the first place, because arbitrary background logic delayed for arbitrary periods of time while other things are being tested is a common source of testing pain, breaking in curious and hard to reproduce ways.

However, @dsnet, who reverted the code, suggested that the decision wasn't necessarily final:

I am personally still in support adding t.Context in conjunction with a wait mechanism as this will counter the current detriment of adding t.Context only. However, I don't believe that we should rush on more API design and we can revisit this for Go 1.9.

I propose that the recently added T.Cleanup method can now provide the wait mechanism that's needed - the other half of Context.Done. We can define the semantics such that the the testing context is cancelled just before invoking the Cleanup-registered functions. That way, it's easy to both listen for the test being done, and also to wait for asynchronous operations to finish when it is.

By hooking into the Cleanup mechanism, we don't presuppose any particular kind of waiting - this can hook into whatever mechanism your infrastructure provides for graceful shutdown.

A possible API description:

// Context returns a context that's cancelled just before
// Cleanup-registered functions are called.
//
// This allows Cleanup-registered functions to wait for any resources
// that are listening on Context.Done before the test completes.
func (t *T) Context() context.Context

Example usage:

func TestFoo(t *testing.T) {
	ctx := t.Context()
	var wg sync.WaitGroup
	t.Cleanup(wg.Wait)
	wg.Add(1)
	go func() {
		defer wg.Done()
		doSomething(ctx)
	}()
}

[seh]

Did you intend for your sample TestFoo function to include a call to WaitGroup.Done?

[rogpeppe]

@seh yup, fixed, thanks!

[rogpeppe]

I'm having second thoughts about this proposal. Given that it's so simple to define your own test-context function, does T.Context actually justify its place?

I believe the function below does almost exactly the same thing as the T.Context method proposed above (the difference being that it returns a new context value every time, but that shouldn't make any significant semantic difference beyond extra memory use):

func testContext(t *testing.T) context.Context {
	ctx, cancel := context.WithCancel(context.Background())
	t.Cleanup(cancel)
	return ctx
}

[bcmills]

I think the changes to golang.org/x/sync/errgroup proposed in CL 134395 are probably cleaner, and wouldn't require any changes to the standard-library testing package.

In particular, that change would allow you to do something like:

func TestFoo(t *testing.T) {
	g, ctx := errgroup.New(context.Background())
	t.Cleanup(g.Stop)
	g.Go(func() error {
		defer wg.Done()
		doSomething(ctx)
		return nil
	})
}

[bcmills]

Thinking about this some more: I think a (*T).Context method would still be too confusing.

Some users would likely assume that it is (only) cancelled if and when the test is marked as failed. Others might assume that it is cancelled at start of the Cleanup phase, before the cleanup functions are invoked. Yet others might assume that it is cancelled after all of the Cleanup functions have completed.

Since there is no single “intuitive” interpretation, we should avoid the ambiguous name. Perhaps we could find a less ambiguous name, but given that the explicit code isn't much longer I'm not sure that's worth the API weight.

[rogpeppe]

After some discussion with @mvdan, I realised that my original proposal here was not sufficient for his use case (he wants to start tearing things down immediately there's a test error).

I don't agree with tearing everything down on any call to Fail, but ISTM that there's an alternative: provide a context that's canceled whenever FailNow is called and we document that it's OK to call FailNow in a goroutine.

Calling FailNow in a goroutine still won't tear down the entire test (it can't, of course) but at least then there's a mechanism whereby that can happen. And people really like calling FailNow in goroutines, and there's no really reason AFAICS why it needs to be disallowed. We could even document that it calls runtime.GoExit under the hood.

wdyt?

[bcmills]

The approach in that errgroup CL still seems more general and a bit cleaner to me.

We don't need to couple cancellation to FailNow; we only need to have a mechanism to tie the call to runtime.Goexit to the cancellation of a corresponding Context, and tying goroutines to higher-level tasks is literally the only purpose of errgroup.Group, so that seems like a natural place for it.

Note that one of the changes in that errgroup CL specifically adds handling and propagation for runtime.Goexit: a runtime.Goexit in any of the associated goroutines cancels the associated Context, and also propagates to any pending or subsequent Wait calls. So one of the effects of that CL is to enable errgroup to tear down (concurrent portions of) a test in the natural way on failure.

[rogpeppe]

I like that thought. The only issue is that you'd have to be careful to add the corresponding code to every goroutine, I guess. That might turn out to be a pain (and it's error-prone).

We'd still need to drop the "FailNow must be called from the goroutine running the test or benchmark function, not from other goroutines created during the test." sentence from the docs. Does that seem reasonable to you?

[bcmills]

We'd still need to drop the "FailNow must be called from the goroutine running the test or benchmark function, not from other goroutines created during the test." sentence from the docs. Does that seem reasonable to you?

That seems reasonable. I would probably replace that sentence with a more explicit one, rather than dropping it outright. Perhaps something like:

FailNow marks the function as having failed and stops its execution the execution of the current goroutine by calling runtime.Goexit (which then runs all of its deferred calls in the current goroutine). Execution will continue at the next test or benchmark. FailNow must be called from the goroutine running the test or benchmark function, not from other goroutines created during the test. Calling FailNow does not stop those other goroutines. FailNow stops only the calling goroutine, not other goroutines associated with or created during the test.

And in the comment for type T:

A test ends when its Test function returns  or calls any of the methods FailNow, Fatal, Fatalf, SkipNow, Skip, or Skipf. Those methods, as well as the Parallel method, must be called only from the goroutine running the Test function. or its goroutine terminates due to a panic or a call to runtime.Goexit. The methods FailNow, Fatal, Fatalf, SkipNow, Skip, and Skipf terminate the calling goroutine, so a call to any of those methods from the goroutine running the Test function ends the test.

And for (*T).Parallel:

Parallel signals that this test is to be run in parallel with (and only with) other parallel tests. When a test is run multiple times due to use of -test.count or -test.cpu, multiple instances of a single test never run in parallel with each other. To ensure correct scheduling, all goroutines associated with the test must block until Parallel returns.

[carnott-snap]

I am interested in continuing this discussion, can we add this to the proposal rotation, or was there something blocking the discussion?

[invidian]

What I've done for couple of projects was to introduce the following function:

const (
  // Arbitrary amount of time to let tests exit cleanly before main process terminates.
  timeoutGracePeriod = 10 * time.Second
)

// contextWithDeadline returns context with will timeout before t.Deadline().
func contextWithDeadline(t *testing.T) context.Context {
  t.Helper()

  deadline, ok := t.Deadline()
  if !ok {
    return context.Background()
  }

  ctx, cancel := context.WithDeadline(context.Background(), deadline.Truncate(timeoutGracePeriod))

  t.Cleanup(cancel)

  return ctx
}

This allows to properly respect -timeout flag in go test. Having a function like this in standard library would be indeed useful, perhaps in a form of:

gracePeriod := time.Second
ctx := t.ContextWithDeadline(gracePeriod)

We could also validate that gracePeriod > now - deadline or alternatively use context.WithTimeout.

[iangudger]

I am strongly in favor of adding this.

It seems like every single test that I write begins with:

func TestXxx(t *testing.T) {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

I have also found that tests which use contexts and don't begin with that preamble usually leak resources or contain other context related bugs. I think making this functionality built-in would help encourage better testing.

[cristaloleg]

Is this proposal in a review? Looks like it's tagged but inactive.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Added this proposal back to minutes due to the ping, but it looks like the conversation meandered quite a bit at the start. It sounds like the current proposal (the only one I see that wasn't recanted) is to have each test basically implicitly start with

func TestXxx(t *testing.T) {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

and then t.Context() returns ctx.

There was some discussion about cancelling the context during FailNow, and about documenting when FailNow is allowed to be called. We can handle the documentation in a separate discussion.

Does anyone think the 'defer cancel' should be replaced / augmented by cancel during FailNow? It seems like FailNow will run the deferred cancel eventually anyway.

It also seems like writing the two lines above is pretty clear and maybe preferable to library magic that you have to guess at when it happens.

It seems like cancel should run before cleanups, so that cleanups can wait for resources, and that happens naturally with the defer as written.

I will try to grep for numbers about how often code in tests uses this pattern explicitly.

[rsc]

I looked into what existing tests do. There are many many with

ctx, cancel := context.WithCancel(context.Background())

but I was surprised how many of those do not do defer cancel().

I extracted one example line from one version of each distinct module where I found a match, shuffled the matches, and posted the results at https://swtch.com/tmp/cancel.html.

Taking the first 25 as a random sample, I found that 15 of them (numbers 2, 5, 6, 10, 11, 13, 14, 15, 16, 18, 20, 22, 23, 24, 25) would not work with an implicit cancel at the end of the test. Instead they are all doing things like canceling midway and then checking that cancelation worked properly.

Given that approximately 60% of the uses of context.WithCancel(context.Background()) would not be helped by adding t.Context, it seems to me that we should probably not add it.

[iangudger]

Another category to consider are tests which use context.Background, but should have used context.WithCancel(context.Background()) and leak resources as a result.

Tests which use context.WithCancel(context.Background()) without a defer and manually call cancel could still wrap t.Context (context.WithCancel(t.Context())) with no ill effect.

[rsc]

@iangudger Hmm, that will be much harder to look for. For example I have written tests recently that use context.Background() and pass that context into various routines, but none of those routines leave any goroutines behind. So it's perfectly fine that I didn't use WithCancel and didn't cancel the context. I suspect that the majority of context.Background() without WithCancel are perfectly fine in this way. I'm reluctant to add new API to fix what may well be a fairly rare problem.

[iangudger]

I'd argue that any test which uses context.Background and fails with context.WithCancel(context.Background()) and a defer at the top is incorrect. As a result, I think it might be reasonable to claim that that most tests which uses an uncancelable context can be counted towards the list of tests which would benefit from a new t.Context.

I've worked at two startups and joined when the companies did not have very many tests. During my tenure at both startups, we wrote a lot of tests. As the number of tests in each package expanded, the issues associated with leaking resources started to show up. At both companies, it happened at least once that running all of the tests in a package would no longer work after adding another test similar to the tests already in the package. Adding a cancelable context with a defer at the beginning of each test fixed the issue at both companies. Further, after converting all of the tests to use cancelable contexts, new tests would occasionally slip in with uncancelable contexts and need to be fixed.

My claim is that:

• All correct tests either benefit from a context canceled at the end of the test or are at worst unaffected by it.
• Not canceling a context at the end of a test commonly causes issues and said issues tend to be difficult to debug.
• ctx, cancel := context.WithCancel(context.Background()); defer cancel() is too cumbersome of a pattern to be commonly used unless you have been bitten by not using it previously. It is also too cumbersome to be consistency used even after you have been bitten by not using it.
• Adding a t.Context method will make using a context which is canceled at the end of the test the path of least resistance for new tests.

[hherman1]

I write too many tests using context.Background() out of laziness, since I’m trying to focus on the rest of the test, and my whole team does the same. t.Context() would be very helpful to us. Ian’s points describe our situation perfectly.

[narqo]

[..] I was surprised how many of those do not do defer cancel()
···
Taking the first 25 as a random sample, I found that 15 of them (numbers 2, 5, 6, 10, 11, 13, 14, 15, 16, 18, 20, 22, 23, 24, 25) would not work with an implicit cancel at the end of the test. Instead they are all doing things like canceling midway and then checking that cancelation worked properly.

I feel that the part about "tests checking that the cancellation works" creates an unintentional bias against the idea of the proposal. That is, such tests have to be explicit about the cancellation, because they validate the behaviour after the cancellation. Those tests could still be written as ctx, cancel := context.WithCancel(t.Context()).

Same time I read the idea behind the proposal as

1. a step to reduce the boilerplate in places where authors do use defer cancel() (or call cancel() in the end of the test function, e.g. 164, 189, 82, etc);
2. a step to encourage authors to always (implicitly) think about the cancellation of the context, in places where context.Background() is used, because of the knowledge of the implementation details (i.e. "nothing in the code path leaves any goroutines behind, so it's fine (today)").

[rsc]

These are good arguments, and it seems in keeping with things like t.Chdir. Does anyone object to adding t.Context back?

@dsnet started this discussion https://groups.google.com/g/golang-dev/c/rS6psxIf17Q back in 2016 when we rolled it back (and issue #18199), which was mainly about not having a way to wait for all the goroutines. Now we have t.Cleanup, and code can make a waitgroup and do wg.Wait as a cleanup. So I think that concern is now addressed and explains why we can now add t.Context back.

[rsc]

Have all remaining concerns about this proposal been addressed?

// Context returns a context that's cancelled just before
// [T.Cleanup]-registered functions are called.
//
// Cleanup functions can wait for any resources
// that shut down on Context.Done before the test completes.
func (t *T) Context() context.Context

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

// Context returns a context that's cancelled just before
// [T.Cleanup]-registered functions are called.
//
// Cleanup functions can wait for any resources
// that shut down on Context.Done before the test completes.
func (t *T) Context() context.Context

[gopherbot]

Change https://go.dev/cl/603959 mentions this issue: testing: add Context

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

// Context returns a context that's cancelled just before
// [T.Cleanup]-registered functions are called.
//
// Cleanup functions can wait for any resources
// that shut down on Context.Done before the test completes.
func (t *T) Context() context.Context