Avoid race in subsequent changes to the same controller

[mborsz]

/kind bug

What this PR does / why we need it:

One of the test runs has failed with

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x17b1e9a]
goroutine 79903 [running]:
k8s.io/perf-tests/clusterloader2/pkg/measurement/common.(*objectChecker).Stop(0x1aa49a0)
	/home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/measurement/common/wait_for_controlled_pods.go:713 +0x7a
k8s.io/perf-tests/clusterloader2/pkg/measurement/util/checker.Map.Add(...)
	/home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/measurement/util/checker/checker_map.go:43
k8s.io/perf-tests/clusterloader2/pkg/measurement/common.(*waitForControlledPodsRunningMeasurement).handleObjectLocked(0xc03c6c9ba0, {0x0, 0x0}, {0x20edda0, 0xc081055010})
	/home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/measurement/common/wait_for_controlled_pods.go:493 +0x1f6
k8s.io/perf-tests/clusterloader2/pkg/measurement/common.(*waitForControlledPodsRunningMeasurement).handleObject(0xc03c6c9ba0, {0x0, 0x0}, {0x1c5b1e0, 0xc081055010})
	/home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/measurement/common/wait_for_controlled_pods.go:453 +0x5b5
k8s.io/perf-tests/clusterloader2/pkg/measurement/common.(*waitForControlledPodsRunningMeasurement).start.func1.1()
	/home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/measurement/common/wait_for_controlled_pods.go:279 +0x2d
k8s.io/perf-tests/clusterloader2/pkg/measurement/util/workerqueue.(*WorkerQueue).worker(0xc02447dba0)
	/home/prow/go/src/k8s.io/perf-tests/clusterloader2/pkg/measurement/util/workerqueue/workerqueue.go:65 +0x31
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.22.15/pkg/util/wait/wait.go:73 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.22.15/pkg/util/wait/wait.go:71 +0x88

This has failed on o.cancel = nil in

perf-tests/clusterloader2/pkg/measurement/common/wait_for_controlled_pods.go

Lines 710 to 713 in 19927c1

	func (o *objectChecker) Stop() {
	o.lock.Lock()
	defer o.lock.Unlock()
	o.cancel()

The cancel is being set in

perf-tests/clusterloader2/pkg/measurement/common/wait_for_controlled_pods.go

Lines 615 to 621 in 83e45df

	o := newObjectChecker(key)
	o.lock.Lock()
	defer o.lock.Unlock()
	w.handlingGroup.Start(func() {
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()
	o.SetCancel(cancel)

The problem is that the setting it happens asynchronously (in the separate goroutine) and when the function finishes there is potential race between calling o.cancel and o.SetCancel.

If the object is being changed between o.SetCancel is called, then the attempt to replace it (and Stop previous instance) will fail as here.

The idea for fix is to:

• make invariant that when the waitForRuntimeObject finishes (so before it's pushed to checkerMap) it has o.cancel set.
• context.WithCancel requires calling cancel as soon as the code finishes to clean resources, so let's keep it where it was (in the async goroutine).

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

/assign @tosi3k

[mborsz]

/assign @marseel

[wojtek-t]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mborsz, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• clusterloader2/OWNERS [mborsz,wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment