Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

teamcity: failed tests on master: test/TestImportControlJob #24623

Closed
cockroach-teamcity opened this issue Apr 10, 2018 · 0 comments
Closed

teamcity: failed tests on master: test/TestImportControlJob #24623

cockroach-teamcity opened this issue Apr 10, 2018 · 0 comments
Assignees
Labels
A-disaster-recovery C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#594979:

--- FAIL: test/TestImportControlJob/cancel (0.000s)
Test ended in panic.

------- Stdout: -------
I180410 02:33:28.046852 21100 sql/event_log.go:121  [n1,client=127.0.0.1:60364,user=root] Event: "create_database", target: 51, info: {DatabaseName:cancelimport Statement:CREATE DATABASE cancelimport User:root}
I180410 02:33:28.047756 21135 storage/replica_command.go:862  [split,n1,s1,r21/1:/{Table/50-Max}] initiating a split of this range at key /Table/51 [r22]
I180410 02:33:28.067322 21162 ccl/importccl/csv.go:232  [import-distsql,n1] could not fetch file size; falling back to per-file progress: bad ContentLength: -1
I180410 02:33:28.095636 21151 storage/replica_command.go:862  [split,n1,s1,r22/1:/{Table/51-Max}] initiating a split of this range at key /Table/52 [r23]
I180410 02:33:36.291426 20767 storage/replica_proposal.go:202  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.290924405,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.292935 20764 storage/replica_proposal.go:202  [n1,s1,r12/1:/Table/1{5-6}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.292649214,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.293679 20771 storage/replica_proposal.go:202  [n1,s1,r11/1:/Table/1{4-5}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.293423875,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.294152 20759 storage/replica_proposal.go:202  [n1,s1,r6/1:/{System/tse-Table/System…}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.293901382,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.294580 20772 storage/replica_proposal.go:202  [n1,s1,r16/1:/Table/{19-20}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.294331389,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.294998 20729 storage/replica_proposal.go:202  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.294747371,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.295420 20713 storage/replica_proposal.go:202  [n1,s1,r9/1:/Table/1{2-3}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.295171574,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.296441 20714 storage/replica_proposal.go:202  [n1,s1,r20/1:/Table/{23-50}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.296144808,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.296879 20773 storage/replica_proposal.go:202  [n1,s1,r15/1:/Table/1{8-9}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.296628125,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.297319 20715 storage/replica_proposal.go:202  [n1,s1,r10/1:/Table/1{3-4}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.297052801,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.297807 20719 storage/replica_proposal.go:202  [n1,s1,r8/1:/Table/1{1-2}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.297570749,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.298306 20716 storage/replica_proposal.go:202  [n1,s1,r19/1:/Table/2{2-3}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.298034425,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.298746 20718 storage/replica_proposal.go:202  [n1,s1,r14/1:/Table/1{7-8}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.298506033,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.299160 20717 storage/replica_proposal.go:202  [n1,s1,r21/1:/Table/5{0-1}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.298915856,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.300204 20721 storage/replica_proposal.go:202  [n1,s1,r23/1:/{Table/52-Max}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.299958547,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.301155 20723 storage/replica_proposal.go:202  [n1,s1,r13/1:/Table/1{6-7}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.300810881,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.302070 20725 storage/replica_proposal.go:202  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.301425825,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.302537 20726 storage/replica_proposal.go:202  [n1,s1,r17/1:/Table/2{0-1}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.302265196,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.302950 20727 storage/replica_proposal.go:202  [n1,s1,r22/1:/Table/5{1-2}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.302707751,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:36.303383 20728 storage/replica_proposal.go:202  [n1,s1,r18/1:/Table/2{1-2}] new range lease repl=(n1,s1):1 seq=3 start=1523327607.732524577,0 epo=1 pro=1523327616.303119505,0 following repl=(n1,s1):1 seq=2 start=1523327607.732524577,0 exp=1523327616.732927797,0 pro=1523327607.733129033,0
I180410 02:33:37.742461 20824 server/status/runtime.go:219  [n1] runtime stats: 243 MiB RSS, 185 goroutines, 16 MiB/65 MiB/102 MiB GO alloc/idle/total, 19 MiB/46 MiB CGO alloc/total, 0.00cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (67x)
I180410 02:33:47.742434 20824 server/status/runtime.go:219  [n1] runtime stats: 245 MiB RSS, 185 goroutines, 12 MiB/69 MiB/102 MiB GO alloc/idle/total, 20 MiB/48 MiB CGO alloc/total, 237.00cgo/sec, 0.03/0.00 %(u/s)time, 0.00 %gc (2x)
I180410 02:33:57.742316 20824 server/status/runtime.go:219  [n1] runtime stats: 245 MiB RSS, 185 goroutines, 17 MiB/66 MiB/102 MiB GO alloc/idle/total, 20 MiB/48 MiB CGO alloc/total, 245.40cgo/sec, 0.09/0.05 %(u/s)time, 0.00 %gc (6x)
I180410 02:34:07.742284 20824 server/status/runtime.go:219  [n1] runtime stats: 245 MiB RSS, 185 goroutines, 13 MiB/69 MiB/102 MiB GO alloc/idle/total, 20 MiB/48 MiB CGO alloc/total, 238.50cgo/sec, 0.58/0.35 %(u/s)time, 0.01 %gc (37x)
I180410 02:34:17.743581 20824 server/status/runtime.go:219  [n1] runtime stats: 245 MiB RSS, 185 goroutines, 14 MiB/68 MiB/102 MiB GO alloc/idle/total, 20 MiB/48 MiB CGO alloc/total, 243.57cgo/sec, 0.54/0.40 %(u/s)time, 0.01 %gc (37x)
I180410 02:34:27.736445 20821 gossip/gossip.go:487  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180410 02:34:27.742333 20824 server/status/runtime.go:219  [n1] runtime stats: 247 MiB RSS, 185 goroutines, 12 MiB/69 MiB/102 MiB GO alloc/idle/total, 21 MiB/50 MiB CGO alloc/total, 242.63cgo/sec, 0.62/0.42 %(u/s)time, 0.01 %gc (39x)
I180410 02:34:37.742659 20824 server/status/runtime.go:219  [n1] runtime stats: 247 MiB RSS, 185 goroutines, 19 MiB/64 MiB/102 MiB GO alloc/idle/total, 21 MiB/50 MiB CGO alloc/total, 238.49cgo/sec, 0.48/0.37 %(u/s)time, 0.00 %gc (33x)
I180410 02:34:47.759554 20824 server/status/runtime.go:219  [n1] runtime stats: 247 MiB RSS, 185 goroutines, 13 MiB/69 MiB/102 MiB GO alloc/idle/total, 21 MiB/50 MiB CGO alloc/total, 236.82cgo/sec, 0.50/0.37 %(u/s)time, 0.01 %gc (33x)
I180410 02:34:57.742697 20824 server/status/runtime.go:219  [n1] runtime stats: 247 MiB RSS, 185 goroutines, 13 MiB/69 MiB/102 MiB GO alloc/idle/total, 21 MiB/50 MiB CGO alloc/total, 240.78cgo/sec, 0.62/0.50 %(u/s)time, 0.01 %gc (35x)
I180410 02:35:07.742376 20824 server/status/runtime.go:219  [n1] runtime stats: 251 MiB RSS, 185 goroutines, 18 MiB/64 MiB/102 MiB GO alloc/idle/total, 21 MiB/51 MiB CGO alloc/total, 244.61cgo/sec, 0.54/0.40 %(u/s)time, 0.01 %gc (35x)
I180410 02:35:17.768326 20824 server/status/runtime.go:219  [n1] runtime stats: 251 MiB RSS, 185 goroutines, 13 MiB/68 MiB/102 MiB GO alloc/idle/total, 21 MiB/51 MiB CGO alloc/total, 237.48cgo/sec, 0.50/0.33 %(u/s)time, 0.01 %gc (36x)
I180410 02:35:27.736445 20821 gossip/gossip.go:487  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180410 02:35:27.744955 20824 server/status/runtime.go:219  [n1] runtime stats: 253 MiB RSS, 185 goroutines, 14 MiB/68 MiB/102 MiB GO alloc/idle/total, 20 MiB/51 MiB CGO alloc/total, 244.07cgo/sec, 0.44/0.28 %(u/s)time, 0.01 %gc (30x)
I180410 02:35:37.760835 20824 server/status/runtime.go:219  [n1] runtime stats: 253 MiB RSS, 185 goroutines, 16 MiB/67 MiB/102 MiB GO alloc/idle/total, 20 MiB/51 MiB CGO alloc/total, 237.22cgo/sec, 0.47/0.35 %(u/s)time, 0.00 %gc (32x)
I180410 02:35:47.745467 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 19 MiB/64 MiB/102 MiB GO alloc/idle/total, 23 MiB/53 MiB CGO alloc/total, 242.07cgo/sec, 0.57/0.41 %(u/s)time, 0.00 %gc (39x)
I180410 02:35:57.749064 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 12 MiB/69 MiB/102 MiB GO alloc/idle/total, 23 MiB/53 MiB CGO alloc/total, 238.41cgo/sec, 0.49/0.40 %(u/s)time, 0.00 %gc (33x)
I180410 02:36:07.742350 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 13 MiB/68 MiB/102 MiB GO alloc/idle/total, 23 MiB/53 MiB CGO alloc/total, 238.66cgo/sec, 0.41/0.31 %(u/s)time, 0.01 %gc (29x)
I180410 02:36:17.742388 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 16 MiB/67 MiB/102 MiB GO alloc/idle/total, 22 MiB/53 MiB CGO alloc/total, 237.30cgo/sec, 0.48/0.36 %(u/s)time, 0.01 %gc (32x)
I180410 02:36:27.736366 20821 gossip/gossip.go:487  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180410 02:36:27.742293 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 17 MiB/66 MiB/102 MiB GO alloc/idle/total, 22 MiB/53 MiB CGO alloc/total, 245.40cgo/sec, 0.47/0.37 %(u/s)time, 0.00 %gc (31x)
I180410 02:36:37.742216 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 17 MiB/66 MiB/102 MiB GO alloc/idle/total, 22 MiB/53 MiB CGO alloc/total, 237.60cgo/sec, 0.42/0.24 %(u/s)time, 0.01 %gc (29x)
I180410 02:36:47.752205 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 14 MiB/68 MiB/102 MiB GO alloc/idle/total, 22 MiB/53 MiB CGO alloc/total, 238.86cgo/sec, 0.41/0.27 %(u/s)time, 0.00 %gc (29x)
I180410 02:36:57.742289 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 14 MiB/68 MiB/102 MiB GO alloc/idle/total, 22 MiB/53 MiB CGO alloc/total, 236.83cgo/sec, 0.39/0.24 %(u/s)time, 0.01 %gc (27x)
I180410 02:37:07.759212 20824 server/status/runtime.go:219  [n1] runtime stats: 255 MiB RSS, 185 goroutines, 17 MiB/66 MiB/102 MiB GO alloc/idle/total, 22 MiB/53 MiB CGO alloc/total, 238.10cgo/sec, 0.49/0.31 %(u/s)time, 0.01 %gc (34x)
panic: test timed out after 4m0s

goroutine 21566 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1240 +0xfc
created by time.goFunc
	/usr/local/go/src/time/sleep.go:172 +0x44

goroutine 1 [chan receive, 3 minutes]:
testing.(*T).Run(0xc423118000, 0x248ac60, 0x14, 0x2594390, 0x8da301)
	/usr/local/go/src/testing/testing.go:825 +0x301
testing.runTests.func1(0xc420116690)
	/usr/local/go/src/testing/testing.go:1063 +0x64
testing.tRunner(0xc420116690, 0xc4206f7dd8)
	/usr/local/go/src/testing/testing.go:777 +0xd0
testing.runTests(0xc4205baca0, 0x3682480, 0xa, 0xa, 0xbeab267a050b4eb0)
	/usr/local/go/src/testing/testing.go:1061 +0x2c4
testing.(*M).Run(0xc42025df80, 0x0)
	/usr/local/go/src/testing/testing.go:978 +0x171
github.com/cockroachdb/cockroach/pkg/ccl/importccl.TestMain(0xc42025df80)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/main_test.go:31 +0xda
main.main()
	_testmain.go:66 +0x151

goroutine 20 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 10 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1128 +0xf1
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x110

goroutine 11 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:598 +0xab
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x128

goroutine 50 [select, 4 minutes, locked to thread]:
runtime.gopark(0x2598e28, 0x0, 0x247342e, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc420015f50, 0xc4200b4240)
	/usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 20521 [chan receive, 3 minutes]:
testing.(*T).Run(0xc423279e00, 0x2472912, 0x6, 0xc422bc9560, 0x14)
	/usr/local/go/src/testing/testing.go:825 +0x301
github.com/cockroachdb/cockroach/pkg/ccl/importccl.TestImportControlJob(0xc423118000)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/csv_test.go:753 +0x3cd
testing.tRunner(0xc423118000, 0x2594390)
	/usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:824 +0x2e0

goroutine 20765 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2eb)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423982070, 0xc4258dc6c0, 0xc423982060)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20767 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2d7)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239820b0, 0xc4258dc6c0, 0xc4239820a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 1873 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420bd37a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab

goroutine 1874 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420bd38c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab

goroutine 83 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420bf38c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab

goroutine 84 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420bf39e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab

goroutine 20782 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x27a36c0, 0xc424f4af60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1503 +0x3ae
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239822a0, 0xc4258dc6c0, 0xc4245d9db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 21187 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fa8a3ea92d0, 0x72, 0xc420079868)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4243df118, 0x72, 0xffffffffffffff00, 0x2785ce0, 0x36967e0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc4243df118, 0xc420711000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4243df100, 0xc420711000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc4243df100, 0xc420711000, 0x1000, 0x1000, 0xc420079978, 0xb7b7fa, 0xc424f15958)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc420d47020, 0xc420711000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6a
net/http.(*connReader).Read(0xc424f15950, 0xc420711000, 0x1000, 0x1000, 0x2, 0x0, 0x2)
	/usr/local/go/src/net/http/server.go:764 +0xf8
bufio.(*Reader).fill(0xc424656480)
	/usr/local/go/src/bufio/bufio.go:100 +0x11e
bufio.(*Reader).ReadSlice(0xc424656480, 0xc4201d0d0a, 0xc420079ad8, 0x85c1d6, 0x7fa8a3dea3d0, 0x0, 0xc420079ac0)
	/usr/local/go/src/bufio/bufio.go:341 +0x2c
bufio.(*Reader).ReadLine(0xc424656480, 0xc423480000, 0x100, 0xf8, 0x243d460, 0x2598501, 0x100c420079b58)
	/usr/local/go/src/bufio/bufio.go:370 +0x34
net/textproto.(*Reader).readLineSlice(0xc424f15980, 0xc422a9f680, 0xc420079ba8, 0x85ceb8, 0x100, 0x243d460)
	/usr/local/go/src/net/textproto/reader.go:55 +0x70
net/textproto.(*Reader).ReadLine(0xc424f15980, 0xc423480000, 0x0, 0x0, 0x72)
	/usr/local/go/src/net/textproto/reader.go:36 +0x2b
net/http.readRequest(0xc424656480, 0x0, 0xc423480000, 0x0, 0x0)
	/usr/local/go/src/net/http/request.go:929 +0x8c
net/http.(*conn).readRequest(0xc422b250e0, 0x27a3600, 0xc423517780, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:944 +0x16f
net/http.(*conn).serve(0xc422b250e0, 0x27a3600, 0xc423517780)
	/usr/local/go/src/net/http/server.go:1768 +0x4dc
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2795 +0x27b

goroutine 20761 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x30d)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271ff0, 0xc4258dc6c0, 0xc424271fa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20615 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x27a36c0, 0xc42422a2d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422fdbd40, 0xc4258dc6c0, 0xc422fdbd30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20616 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x27a36c0, 0xc42422a300)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x162
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422fdbd60, 0xc4258dc6c0, 0xc422fdbd50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 21133 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7fa8a3ea9540, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc423427498, 0x72, 0xc421f06b00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc423427498, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc423427480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:372 +0x1a8
net.(*netFD).accept(0xc423427480, 0x8ad1d7, 0xc4223b64e0, 0x2177000)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4252ce790, 0x8a33a0, 0xc421c48ef0, 0xc421c48ef8)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4252ce790, 0x2598750, 0xc4223b6460, 0x27a36c0, 0xc424977980)
	/usr/local/go/src/net/tcpsock.go:259 +0x49
net/http.(*Server).Serve(0xc422bd3c70, 0x27a1800, 0xc4252ce790, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2770 +0x1a5
net/http/httptest.(*Server).goServe.func1(0xc423180690)
	/usr/local/go/src/net/http/httptest/server.go:280 +0x6d
created by net/http/httptest.(*Server).goServe
	/usr/local/go/src/net/http/httptest/server.go:278 +0x5c

goroutine 1099 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4211b0a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab

goroutine 20786 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x27a36c0, 0xc424f4b0b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:575 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4233dc0d0, 0xc4258dc6c0, 0xc4236e4200)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20974 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc4204a8a00, 0xc423dd67d0, 0xc422682480, 0xf, 0xc42395d6e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:587 +0x3f1
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1(0x27a36c0, 0xc423892a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:507 +0x74
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4225238c0, 0xc4258dc6c0, 0xc423892a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 21097 [select, 3 minutes]:
database/sql.(*DB).connectionResetter(0xc420d08140, 0x27a3600, 0xc42287c680)
	/usr/local/go/src/database/sql/sql.go:948 +0x12a
created by database/sql.OpenDB
	/usr/local/go/src/database/sql/sql.go:635 +0x1ae

goroutine 20770 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2d5)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423982110, 0xc4258dc6c0, 0xc423982100)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20774 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2ff)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a4e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a4e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423982190, 0xc4258dc6c0, 0xc423982180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20719 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2e3)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4242718d0, 0xc4258dc6c0, 0xc4242718c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20778 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc422bcdb00, 0x27a36c0, 0xc424f4ab40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4014 +0x1e7
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.raftTickLoop)-fm(0x27a36c0, 0xc424f4ab40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4000 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423982210, 0xc4258dc6c0, 0xc423982200)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20712 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x27a36c0, 0xc424f4a6f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:156 +0x49
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4242717e0, 0xc4258dc6c0, 0xc422764320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20639 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x27a36c0, 0xc42422a7b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1160 +0x418
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424270330, 0xc4258dc6c0, 0xc424270320)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20991 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420895200)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1323 +0x137
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:309 +0xdb5

goroutine 20597 [select, 3 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc420540500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1234 +0x235
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc420540500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:837 +0x1b5
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:828 +0xe1

goroutine 20749 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x304)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a1b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a1b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271d70, 0xc4258dc6c0, 0xc424271d60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20783 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x27a36c0, 0xc424f4af90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1503 +0x3ae
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239822b0, 0xc4258dc6c0, 0xc4245d9e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20755 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x308)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a2d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a2d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271ed0, 0xc4258dc6c0, 0xc424271ec0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20788 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x27a36c0, 0xc424f4b110)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:575 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4233dc0f0, 0xc4258dc6c0, 0xc4236e4540)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 1098 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4211b0900)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab

goroutine 20630 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc424a1c6e0, 0xc4247b02a0, 0xc421101688, 0x9728ca, 0x233f9a0)
	<autogenerated>:1 +0x5b
crypto/tls.(*listener).Accept(0xc424a1c720, 0xc4200ca050, 0x218efe0, 0x366c800, 0x242e520)
	/usr/local/go/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc421c9e8f0, 0x2792f40, 0xc424a1c720, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2770 +0x1a5
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func5(0x27a36c0, 0xc4247b0240)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1059 +0x42
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424270010, 0xc4258dc6c0, 0xc424a1c740)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 1636 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420956000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab

goroutine 1637 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420956120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1180 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x2ab

goroutine 20750 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x305)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a1e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a1e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271dd0, 0xc4258dc6c0, 0xc424271dc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20780 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x27a36c0, 0xc424f4aed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1369 +0x10e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423982280, 0xc4258dc6c0, 0xc422764340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20754 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x30c)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271eb0, 0xc4258dc6c0, 0xc424271ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20779 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc422bcdb00, 0x27a36c0, 0xc424f4aa80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4055 +0x159
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x27a36c0, 0xc424f4aa80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:4001 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423982240, 0xc4258dc6c0, 0xc423982230)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20777 [semacquire, 3 minutes]:
sync.runtime_Semacquire(0xc421542744)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc421542738)
	/usr/local/go/src/sync/waitgroup.go:129 +0x72
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc4215426c0, 0x27a36c0, 0xc424f4ab10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x31
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x27a36c0, 0xc424f4ab10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3998 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4239821f0, 0xc4258dc6c0, 0xc4239821e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20751 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x302)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271e10, 0xc4258dc6c0, 0xc424271df0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20758 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x30e)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271f50, 0xc4258dc6c0, 0xc424271f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20752 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x307)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271e30, 0xc4258dc6c0, 0xc424271e20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20753 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x309)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271e90, 0xc4258dc6c0, 0xc424271e50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20522 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc4258dc6c0, 0xc4226bc150)
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:122 +0x108
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:119 +0x49

goroutine 20789 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x27a36c0, 0xc424f4b140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:575 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4233dc100, 0xc4258dc6c0, 0xc4236e4560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20716 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2e1)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a7b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a7b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271860, 0xc4258dc6c0, 0xc424271850)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20555 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.loopyWriter(0x27a36c0, 0xc42422a750, 0xc42422a6f0, 0xc42093ffb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:727 +0x368
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc4256aa000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:261 +0x5e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:260 +0xe08

goroutine 20726 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2e7)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a9f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a9f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271a50, 0xc4258dc6c0, 0xc424271a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20781 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x27a36c0, 0xc424f4ae70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1503 +0x3ae
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423982290, 0xc4258dc6c0, 0xc4245d9d60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20720 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2e4)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271900, 0xc4258dc6c0, 0xc4242718e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20635 [select, 3 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc424a1b680)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:110 +0x182
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:96 +0x3f

goroutine 20787 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x27a36c0, 0xc424f4b0e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:575 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4233dc0e0, 0xc4258dc6c0, 0xc4236e4220)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20741 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2fb)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4ade0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4ade0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271c30, 0xc4258dc6c0, 0xc424271c20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20557 [select]:
net.(*pipe).read(0xc42085c800, 0xc423206000, 0x400, 0x400, 0x8, 0x8, 0x3f3)
	/usr/local/go/src/net/pipe.go:164 +0x1d7
net.(*pipe).Read(0xc42085c800, 0xc423206000, 0x400, 0x400, 0x235fd00, 0x0, 0x7fa8a3ebc370)
	/usr/local/go/src/net/pipe.go:147 +0x4d
crypto/tls.(*block).readFromUntil(0xc42422a570, 0x7fa8a3ebc370, 0xc42085c800, 0x5, 0xc42085c800, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x96
crypto/tls.(*Conn).readRecord(0xc42376e000, 0x2598f17, 0xc42376e120, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:595 +0xe0
crypto/tls.(*Conn).Read(0xc42376e000, 0xc42353a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x100
bufio.(*Reader).Read(0xc420d4e000, 0xc420518498, 0x9, 0x9, 0x20, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:216 +0x238
io.ReadAtLeast(0x277fb80, 0xc420d4e000, 0xc420518498, 0x9, 0x9, 0x9, 0xc421b13d88, 0x84c16e, 0xc421b13e37)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x277fb80, 0xc420d4e000, 0xc420518498, 0x9, 0x9, 0xcedc2d, 0xc42426802c, 0xc423361c90)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420518498, 0x9, 0x9, 0x277fb80, 0xc420d4e000, 0x0, 0x0, 0xc424268020, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420518460, 0xc424268020, 0xc424268020, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4256aa000, 0xc424f4a060, 0x25981d0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:395 +0x37
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc421fa22c0, 0x27b4660, 0xc4256aa000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:634 +0xeb
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:591 +0x3c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4236e4000, 0xc421fa22c0, 0x7fa8a3dcf4a8, 0xc4256aa000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:599 +0x27
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:598 +0x5f9

goroutine 20748 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x301)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a180)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271d50, 0xc4258dc6c0, 0xc424271d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20821 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x27a36c0, 0xc424f4ba40)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:706 +0x2b5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423982600, 0xc4258dc6c0, 0xc422764580)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20725 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2e6)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4a9c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4a9c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271a30, 0xc4258dc6c0, 0xc424271a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20632 [chan receive, 3 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc424a1c640, 0x2598188, 0xc421fa22c0, 0x27b6220, 0xc420e6fe00)
	<autogenerated>:1 +0x5b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc421fa22c0, 0x27a2380, 0xc424a1c640, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:504 +0x1d5
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x27a36c0, 0xc42422a4e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1075 +0x43
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424270030, 0xc4258dc6c0, 0xc424a1c760)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20737 [semacquire]:
sync.runtime_notifyListWait(0xc423f3a7d0, 0x2f9)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc423f3a7c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4215426c0, 0x27a36c0, 0xc424f4ad20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x27a36c0, 0xc424f4ad20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424271bb0, 0xc4258dc6c0, 0xc424271ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:185 +0xad

goroutine 20561 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x27a36c0, 0xc424f4b080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storag

Please assign, take a look and update the issue accordingly.

@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone Apr 10, 2018
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Apr 10, 2018
craig bot pushed a commit that referenced this issue Oct 22, 2018
31556: importccl: re-enable job control tests r=mjibson a=mjibson

I tracked down the problem. It was that after the CANCEL JOB was issued,
sometimes the 2nd stage of the IMPORT (the shuffle) would have started,
and sometimes it wouldn't have. If it did not start then RunJob would
block forever trying to send on the allowResponse chan. Fix this by
making a draining go routine after the first block.

Closes #24623
Closes #24658

Release note: None

31627: storage: remove spurious call to maybeInlineSideloadedRaftCommand r=nvanbenschoten,benesch a=tschottdorf

Entries are "thinned" only when passed to `r.append()` (i.e. written to
disk) and they are always returned "fat" from `Entries()` (i.e. Raft's way
to get entries from disk). Consequently Raft never sees thin entries and
won't ask us to commit them.

Touches #31618.

Release note: None

31695: bitarray: don't allow FromEncodingParts to return invalid bit array r=knz a=benesch

It is invalid for a bit array's lastBitsUsed field to be greater than
64. The FromEncodingParts function, however, would happily construct
an invalid bitarray if given a too-large lastBitsUsed value. Teach the
FromEncodingParts to return an error instead.

This presented as a panic when attempting to pretty-print a key with a
bitarray whose lastBitsUsed encoded value was 65. Such a key can be
created when calling PrefixEnd on a key with a bitarray whose
lastBitsUsed value is 64. By returning an error instead, the
pretty-printing code will try again after calling UndoPrefixEnd and be
able to print the key.

Fix #31115.

Release note: None

31697: partitionccl: deflake TestDropIndexWithZoneConfigCCL r=danhhz,eriktrinh a=benesch

A particularly adversarial goroutine schedule can cause this test to
observe the moment in time where the data is dropped but the zone config
is not. Deflake by retrying the check for the dropped zone config until
it succeeds (or times out).

Fix #31678.

Release note: None

Co-authored-by: Matt Jibson <matt.jibson@gmail.com>
Co-authored-by: Tobias Schottdorf <tobias.schottdorf@gmail.com>
Co-authored-by: Nikhil Benesch <nikhil.benesch@gmail.com>
@craig craig bot closed this as completed in #31556 Oct 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-disaster-recovery C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

No branches or pull requests

3 participants