goleak at the structure

[hawkingrei]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

goleak: Errors on successful test run: found unexpected goroutines:
[Goroutine 27268 in state select, with github.com/pingcap/tidb/ttl/ttlworker.(*ttlScanWorker).loop on top of the stack:
goroutine 27268 [select]:
github.com/pingcap/tidb/ttl/ttlworker.(*ttlScanWorker).loop(0xc00759a510)
	ttl/ttlworker/scan.go:315 +0x25f
github.com/pingcap/tidb/ttl/ttlworker.(*baseWorker).loop(0xc00759a510)
	ttl/ttlworker/worker.go:134 +0xa4
github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1()
	util/wait_group_wrapper.go:154 +0x74
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run
	util/wait_group_wrapper.go:152 +0xe5
 Goroutine 27266 in state chan receive, with github.com/pingcap/tidb/ttl/client.(*mockClient).WatchCommand.func1 on top of the stack:
goroutine 27266 [chan receive]:
github.com/pingcap/tidb/ttl/client.(*mockClient).WatchCommand.func1()
	ttl/client/command.go:424 +0x7f
created by github.com/pingcap/tidb/ttl/client.(*mockClient).WatchCommand
	ttl/client/command.go:423 +0x4ea
 Goroutine 27267 in state select, with github.com/pingcap/tidb/ttl/ttlworker.(*ttlScanWorker).loop on top of the stack:
goroutine 27267 [select]:
github.com/pingcap/tidb/ttl/ttlworker.(*ttlScanWorker).loop(0xc00759a480)
	ttl/ttlworker/scan.go:315 +0x25f
github.com/pingcap/tidb/ttl/ttlworker.(*baseWorker).loop(0xc00759a480)
	ttl/ttlworker/worker.go:134 +0xa4
github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1()
	util/wait_group_wrapper.go:154 +0x74
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run
	util/wait_group_wrapper.go:152 +0xe5
 Goroutine 27159 in state select, with github.com/pingcap/tidb/ttl/ttlworker.(*JobManager).jobLoop on top of the stack:
goroutine 27159 [select]:
github.com/pingcap/tidb/ttl/ttlworker.(*JobManager).jobLoop(0xc004b4bc20)
	ttl/ttlworker/job_manager.go:171 +0x8f4
github.com/pingcap/tidb/ttl/ttlworker.(*baseWorker).loop(0xc004b4bc20)
	ttl/ttlworker/worker.go:134 +0xa4
github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1()
	util/wait_group_wrapper.go:154 +0x74
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run
	util/wait_group_wrapper.go:152 +0xe5
 Goroutine 27270 in state select, with github.com/pingcap/tidb/ttl/ttlworker.(*ttlScanWorker).loop on top of the stack:
goroutine 27270 [select]:
github.com/pingcap/tidb/ttl/ttlworker.(*ttlScanWorker).loop(0xc00759a630)
	ttl/ttlworker/scan.go:315 +0x25f
github.com/pingcap/tidb/ttl/ttlworker.(*baseWorker).loop(0xc00759a630)
	ttl/ttlworker/worker.go:134 +0xa4
github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1()
	util/wait_group_wrapper.go:154 +0x74
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run
	util/wait_group_wrapper.go:152 +0xe5
 Goroutine 27269 in state select, with github.com/pingcap/tidb/ttl/ttlworker.(*ttlScanWorker).loop on top of the stack:
goroutine 27269 [select]:
github.com/pingcap/tidb/ttl/ttlworker.(*ttlScanWorker).loop(0xc00759a5a0)
	ttl/ttlworker/scan.go:315 +0x25f
github.com/pingcap/tidb/ttl/ttlworker.(*baseWorker).loop(0xc00759a5a0)
	ttl/ttlworker/worker.go:134 +0xa4
github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run.func1()
	util/wait_group_wrapper.go:154 +0x74
created by github.com/pingcap/tidb/util.(*WaitGroupWrapper).Run
	util/wait_group_wrapper.go:152 +0xe5
]
================================================================================

https://prow.hawkingrei.com/view/gs/prowpingcap/logs/tidb_race/1646640605893234688

4. What is your TiDB version? (Required)

[YangKeao]

I think it's an edge case in which the domain starts and stops too fast. The following logs showed that the ttl starts to scale up the worker after the context reached the end:

[2023/04/14 08:09:33.631 +08:00] [INFO] [domain.go:2699] ["ttlJobManager exited."]
[2023/04/14 08:09:33.631 +08:00] [INFO] [wait_group_wrapper.go:137] ["background process exited"] [source=domain] [process=ttlJobManager]
[2023/04/14 08:09:33.632 +08:00] [INFO] [domain.go:997] ["domain closed"] ["take time"=21.507792ms]
[2023/04/14 08:09:33.634 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]
[2023/04/14 08:09:33.634 +08:00] [INFO] [task_manager.go:215] ["scale ttl worker"] [ttl-worker=job-manager] [ttl-worker=task-manager] [originalCount=0] [newCount=4]

But It's still quite amazing that the ttl workers are still working after domain closed, and I haven't found the accurate problem.

[YangKeao]

I'll add more log and try to reproduce at local machine 🤔, hope to get more clues.

[zimulala]

Reproduce link: https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftidb%2Fghpr_unit_test/detail/ghpr_unit_test/22777/pipeline