Coroutine leak after kill remote connection

[pingyu]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Setup a cluster with two TiDB.
2. From one of the TiDB, kill connection on another one repeatedly.

2. What did you expect to see? (Required)

Nothing wrong.

3. What did you see instead (Required)

Goroutine count keep increasing, and OOM at last.

Goroutine profiling:

goroutine profile: total 759643
379246 @ 0x1abd2f6 0x1acf0cf 0x1acf0a6 0x1aee9a7 0x1b02f8b 0x33539f7 0x1af3301
# labels: {"plan_digest":"", "sql_digest":"W\x8b\x99s\xf5\xfb\x0f\xe4/\xf8\xec_\xa0\xc75\xa3R,\xbc(\x8f\xaeV&\xacv\xb4B*S\x97+"}
#	0x1aee9a6	sync.runtime_Semacquire+0x26						/usr/local/go/src/runtime/sema.go:62
#	0x1b02f8a	sync.(*WaitGroup).Wait+0x4a						/usr/local/go/src/sync/waitgroup.go:116
#	0x33539f6	github.com/pingcap/tidb/store/copr.(*copIteratorTaskSender).run+0x1d6	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:901

379236 @ 0x1abd2f6 0x1acde1e 0x335400d 0x33577f4 0x3356098 0x33549af 0x3353168 0x1af3301
# labels: {"plan_digest":"", "sql_digest":"W\x8b\x99s\xf5\xfb\x0f\xe4/\xf8\xec_\xa0\xc75\xa3R,\xbc(\x8f\xaeV&\xacv\xb4B*S\x97+"}
#	0x335400c	github.com/pingcap/tidb/store/copr.(*copIteratorWorker).sendToRespCh+0xac	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:992
#	0x33577f3	github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleCopResponse+0x7b3	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1407
#	0x3356097	github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTaskOnce+0x10d7	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1251
#	0x33549ae	github.com/pingcap/tidb/store/copr.(*copIteratorWorker).handleTask+0x18e	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:1119
#	0x3353167	github.com/pingcap/tidb/store/copr.(*copIteratorWorker).run+0xc7		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/store/copr/coprocessor.go:806

776 @ 0x1abd2f6 0x1ab5517 0x1aec4c9 0x1b7b9d2 0x1b7cf39 0x1b7cf27 0x1c96789 0x1caad45 0x1d47d5b 0x4c47e70 0x1b730ba 0x4c4a99f 0x4c4a758 0x4c4ad78 0x4cb4605 0x4cb88f9 0x4cdb2a7 0x1af3301
# labels: {"plan_digest":"", "sql_digest":"\xf4\xb7ԓ \xa9\xf4\xeb\xce\x10\xb1ʺ\x19\xd8\xe30\x05\x11\x86\xdd=\\=\x80\xa3*\xc1\x8c\x12\xb4A"}
#	0x1aec4c8	internal/poll.runtime_pollWait+0x88					/usr/local/go/src/runtime/netpoll.go:306
#	0x1b7b9d1	internal/poll.(*pollDesc).wait+0x31					/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0x1b7cf38	internal/poll.(*pollDesc).waitRead+0x298				/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#	0x1b7cf26	internal/poll.(*FD).Read+0x286						/usr/local/go/src/internal/poll/fd_unix.go:167
#	0x1c96788	net.(*netFD).Read+0x28							/usr/local/go/src/net/fd_posix.go:55
#	0x1caad44	net.(*conn).Read+0x44							/usr/local/go/src/net/net.go:183
#	0x1d47d5a	bufio.(*Reader).Read+0x1ba						/usr/local/go/src/bufio/bufio.go:237
#	0x4c47e6f	github.com/pingcap/tidb/server/internal/util.BufferedReadConn.Read+0x2f	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/internal/util/buffered_read_conn.go:41
#	0x1b730b9	io.ReadAtLeast+0x99							/usr/local/go/src/io/io.go:332
#	0x4c4a99e	io.ReadFull+0x39e							/usr/local/go/src/io/io.go:351
#	0x4c4a757	github.com/pingcap/tidb/server/internal.(*PacketIO).readOnePacket+0x157	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/internal/packetio.go:179
#	0x4c4ad77	github.com/pingcap/tidb/server/internal.(*PacketIO).ReadPacket+0x57	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/internal/packetio.go:227
#	0x4cb4604	github.com/pingcap/tidb/server.(*clientConn).readPacket+0x64		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:434
#	0x4cb88f8	github.com/pingcap/tidb/server.(*clientConn).Run+0x218			/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1008
#	0x4cdb2a6	github.com/pingcap/tidb/server.(*Server).onConn+0x806			/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:699

51 @ 0x1abd2f6 0x1ab5517 0x1aec4c9 0x1b7b9d2 0x1b7cf39 0x1b7cf27 0x1c96789 0x1caad45 0x1d47d5b 0x4c47e70 0x1b730ba 0x4c4a99f 0x4c4a758 0x4c4ad78 0x4cb4605 0x4cb88f9 0x4cdb2a7 0x1af3301
# labels: {"plan_digest":"", "sql_digest":"W\x8b\x99s\xf5\xfb\x0f\xe4/\xf8\xec_\xa0\xc75\xa3R,\xbc(\x8f\xaeV&\xacv\xb4B*S\x97+"}
#	0x1aec4c8	internal/poll.runtime_pollWait+0x88					/usr/local/go/src/runtime/netpoll.go:306
#	0x1b7b9d1	internal/poll.(*pollDesc).wait+0x31					/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0x1b7cf38	internal/poll.(*pollDesc).waitRead+0x298				/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#	0x1b7cf26	internal/poll.(*FD).Read+0x286						/usr/local/go/src/internal/poll/fd_unix.go:167
#	0x1c96788	net.(*netFD).Read+0x28							/usr/local/go/src/net/fd_posix.go:55
#	0x1caad44	net.(*conn).Read+0x44							/usr/local/go/src/net/net.go:183
#	0x1d47d5a	bufio.(*Reader).Read+0x1ba						/usr/local/go/src/bufio/bufio.go:237
#	0x4c47e6f	github.com/pingcap/tidb/server/internal/util.BufferedReadConn.Read+0x2f	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/internal/util/buffered_read_conn.go:41
#	0x1b730b9	io.ReadAtLeast+0x99							/usr/local/go/src/io/io.go:332
#	0x4c4a99e	io.ReadFull+0x39e							/usr/local/go/src/io/io.go:351
#	0x4c4a757	github.com/pingcap/tidb/server/internal.(*PacketIO).readOnePacket+0x157	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/internal/packetio.go:179
#	0x4c4ad77	github.com/pingcap/tidb/server/internal.(*PacketIO).ReadPacket+0x57	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/internal/packetio.go:227
#	0x4cb4604	github.com/pingcap/tidb/server.(*clientConn).readPacket+0x64		/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:434
#	0x4cb88f8	github.com/pingcap/tidb/server.(*clientConn).Run+0x218			/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1008
#	0x4cdb2a6	github.com/pingcap/tidb/server.(*Server).onConn+0x806			/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:699

...
...

4. What is your TiDB version? (Required)

master (92b5a67)

[pingyu]

/assign