Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock in SHOW TABLES #44385

Closed
mrigger opened this issue Jan 25, 2020 · 7 comments
Closed

Deadlock in SHOW TABLES #44385

mrigger opened this issue Jan 25, 2020 · 7 comments
Assignees
@thoszhang
Labels
A-schema-descriptors Relating to SQL table/db descriptor handling. A-sql-vtables Virtual tables - pg_catalog, information_schema etc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting.

Comments

@mrigger
Copy link

mrigger commented Jan 25, 2020 •
edited
Loading

It seems that SHOW TABLES in combination with other concurrent requests can result in a deadlock. This can be reproduced by executing these commands, as demonstrated below:

cockroach sql --watch '1ms' -e 'DROP DATABASE IF EXISTS db0;CREATE DATABASE db0;USE db0;CREATE TABLE t0(c0 INT);'
cockroach sql --watch '1ms' -e 'SHOW TABLES;'

I found this issue using 2c4c0f92b8d137054e03872e6503561465f55eee.

UPDATE: While I initially reported this issue privately, I now included all the information as part of this bug report, since the issue does not seem to be security-relevant.
@yuzefovich yuzefovich added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Jan 26, 2020
@yuzefovich yuzefovich added the S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting. label Jan 26, 2020
@solongordon solongordon assigned thoszhang and unassigned solongordon Jan 28, 2020
@knz knz added A-schema-descriptors Relating to SQL table/db descriptor handling. A-sql-vtables Virtual tables - pg_catalog, information_schema etc labels Feb 1, 2020
@mrigger
Copy link
Author

mrigger commented Feb 9, 2020

FYI: This issue is slowing down my testing efforts, since I can only use a single thread to test CockroachDB.

@otan
Copy link
Contributor

otan commented Feb 12, 2020 •
edited
Loading

I'm looking at the goroutine dump.

looks like the drop_table.go deadlocks on the storage txn wait layer


goroutine 1451 [select]:
github.com/cockroachdb/cockroach/pkg/storage/txnwait.(*Queue).MaybeWaitForPush(0xc000840600, 0x824efa0, 0xc005b69cb0, 0xc003879c20, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/txnwait/queue.go:506 +0xfe8
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).maybeWaitForPushee(0xc000bf0000, 0x824efa0, 0xc005b69cb0, 0xc005b2cc80, 0xc005ce97a0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_send.go:253 +0xb6
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeBatchWithConcurrencyRetries(0xc000bf0000, 0x824efa0, 0xc005b69cb0, 0xc005b2cc80, 0x7b6f308, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:185 +0x19d
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc000bf0000, 0x824efa0, 0xc005b69c80, 0x6, 0xc005b2cc80, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:95 +0x68a
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc000bf0000, 0x824efa0, 0xc005b69c80, 0x15f2c29bbf063968, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:37 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000a9d500, 0x824efa0, 0xc005b69c20, 0x15f2c29bbf063968, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc000900140, 0x824efa0, 0xc005b69c20, 0x15f2c29bbf063968, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x824efa0, 0xc005b69c20, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:935 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0009d0ab0, 0x824efa0, 0xc005b69c20, 0x79e67ff, 0x10, 0xc006d343a0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:301 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000150a00, 0x824efa0, 0xc005b69c20, 0xc005b2cc00, 0xc005b69c20, 0xc005b699e0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:923 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000150a00, 0x824efa0, 0xc005b69bc0, 0xc005b2cc00, 0xc005b69b00, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:961 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc004c9ae80, 0x824efa0, 0xc005b69bc0, 0x1, 0x8210ba0, 0xc00030f0e0, 0x15f2c29bbf063968, 0x0, 0x100000001, 0x1, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc004c9ae80, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000318240, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1592 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000318240, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:440 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000318240, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:522 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000318240, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1363 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000318240, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1058 +0x146f
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000318240, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:708 +0x7b4
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0009a3860, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:223 +0x95
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0009a3800, 0x824efa0, 0xc005b69b00, 0x15f2c29bbf063968, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:736
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x824efa0, 0xc005b69b00, 0xc006d36050, 0xc006283080, 0xc006d36738, 0xc006d362b0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0009a3800, 0x824efa0, 0xc005b69b00, 0xc006283080, 0x19, 0xc006d36280)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:695 +0x9c
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).MaybePushTransactions(0xc0002ba540, 0x824efa0, 0xc005b69b00, 0xc006d36738, 0x15f2c29bbf056e48, 0x0, 0x100000001, 0x1, 0x0, 0x1a, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:444 +0x760
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).maybePushIntents(0xc0002ba540, 0x824efa0, 0xc005b69b00, 0xc00363fe40, 0x1, 0x1, 0x15f2c29bbf056e48, 0x0, 0x100000001, 0x1, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:348 +0x279
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).ProcessWriteIntentError(0xc0002ba540, 0x824efa0, 0xc005b69b00, 0xc0062cee10, 0x15f2c29bbf056e48, 0x0, 0x100000001, 0x1, 0x0, 0x1a, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:265 +0x147
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleWriteIntentError(0xc000a14800, 0x824efa0, 0xc005b69b00, 0xc005b2cb00, 0xc0062cee10, 0xc00595c7a0, 0x0, 0xc0062cee10, 0xc006d36ff8)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:281 +0x14b
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeBatchWithConcurrencyRetries(0xc000a14800, 0x824efa0, 0xc005b69b00, 0xc005b2cb00, 0x7b6f2e8, 0x0, 0xc0062cee10)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:205 +0x40f
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc000a14800, 0x824efa0, 0xc005b69ad0, 0x1a, 0xc005b2cb00, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:99 +0x2f3
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc000a14800, 0x824efa0, 0xc005b69ad0, 0x15f2c29bbf056e48, 0x0, 0x100000001, 0x1, 0x0, 0x1a, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:37 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000a9d500, 0x824efa0, 0xc005b69a70, 0x15f2c29bbf056e48, 0x0, 0x100000001, 0x1, 0x0, 0x1a, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc000900140, 0x824efa0, 0xc005b69a70, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x1a, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x824efa0, 0xc005b69a70, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:935 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0009d0ab0, 0x824efa0, 0xc005b69a70, 0x79e67ff, 0x10, 0xc006d37cb0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:301 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000150a00, 0x824efa0, 0xc005b69a70, 0xc005b2ca80, 0xc005b69a70, 0x80, 0x80)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:923 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000150a00, 0x824efa0, 0xc005b69a40, 0xc005b2ca80, 0xc005b699e0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:961 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc004c9ac80, 0x824efa0, 0xc005b69a40, 0x1, 0x8210ba0, 0xc00030f0e0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc004c9ac80, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x1a, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000318240, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1a, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1592 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000318240, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1a, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:440 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000318240, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:522 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000318240, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1363 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000318240, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1058 +0x146f
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000318240, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:708 +0x7b4
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc00644ff08, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetricRecorder).SendLocked(0xc00644fed0, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metric_recorder.go:46 +0x8d
github.com/cockroachdb/cockroach/pkg/kv.(*txnCommitter).SendLocked(0xc00644fea0, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_committer.go:126 +0x6f9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendHelper(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:345
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc00644fe38, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:230 +0xa9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc00644fe38, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:179 +0x158
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc00644fd88, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:208 +0x159
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc00644fd68, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).SendLocked(0xc00644fcc8, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:168 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc00644fb00, 0x824efa0, 0xc005b699e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:488 +0x406
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0009a3800, 0x824efa0, 0xc005b68f60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc005d5d170, 0x824efa0, 0xc005b68f60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:870 +0x11e
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x824efa0, 0xc005b68f60, 0xc006d3a630, 0xc006282b00, 0x1, 0x1)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc005d5d170, 0x824efa0, 0xc005b68f60, 0xc006282b00, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:538 +0xfa
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Get(0xc005d5d170, 0x824efa0, 0xc005b68f60, 0x774cc80, 0xc00595c6c0, 0xc006d3a870, 0x1, 0x1, 0xc006d3a850, 0xc00010fee0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:348 +0xbc
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.LookupObjectID(0x824efa0, 0xc005b68f60, 0xc005d5d170, 0x0, 0xc0064e2690, 0x3, 0xc00644fe38, 0x82904c0, 0xc00644fea0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/namespace.go:164 +0x1cb
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.LookupDatabaseID(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/namespace.go:214
github.com/cockroachdb/cockroach/pkg/sql.getDatabaseID(0x824efa0, 0xc005b68f60, 0xc005d5d170, 0xc0064e2690, 0x3, 0x0, 0x0, 0xc006d3a9c8, 0x41f403c)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:82 +0x7e
github.com/cockroachdb/cockroach/pkg/sql.(*databaseCache).getDatabaseID.func1(0x824efa0, 0xc005b68f60, 0xc005d5d170, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:226 +0x62
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x824efa0, 0xc005b68f60, 0xc005d5d170, 0x82dcf40, 0xc00644fb00)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:717 +0x43
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).exec(0xc005d5d170, 0x824efa0, 0xc005b68f60, 0xc007034d50, 0xc005d5d170, 0x28)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:759 +0xd9
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc0009a3800, 0x824efa0, 0xc005b68f60, 0xc005b699b0, 0x1, 0xc005b699b0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:716 +0xd1
github.com/cockroachdb/cockroach/pkg/sql.(*databaseCache).getDatabaseID(0xc0057ec6c0, 0x824efa0, 0xc005b68f60, 0xc006d3ae88, 0xc0064e2690, 0x3, 0x0, 0xc000304b40, 0x0, 0xc006d3aef0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:224 +0x19d
github.com/cockroachdb/cockroach/pkg/sql.(*TableCollection).getMutableTableDescriptor(0xc005483238, 0x824efa0, 0xc005b68f60, 0xc006ca2fc0, 0xc0054838a0, 0x10000, 0x0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:204 +0x48f
github.com/cockroachdb/cockroach/pkg/sql.(*CachedPhysicalAccessor).GetObjectDesc(0xc0054a0060, 0x824efa0, 0xc005b68f60, 0xc006ca2fc0, 0xc00052ca80, 0xc0054838a0, 0x10000, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/physical_schema_accessors.go:302 +0x96
github.com/cockroachdb/cockroach/pkg/sql.(*LogicalSchemaAccessor).GetObjectDesc(0xc0054a00a0, 0x824efa0, 0xc005b68f60, 0xc006ca2fc0, 0xc00052ca80, 0xc0054838a0, 0x10000, 0x6, 0x101, 0xc006d3bc00, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/logical_schema_accessors.go:100 +0x65e
github.com/cockroachdb/cockroach/pkg/sql.(*planner).LookupObject(0xc0054833c0, 0x824efa0, 0xc005b68f60, 0x10000, 0xc0064e2690, 0x3, 0x79cc097, 0x6, 0xc004345a1e, 0x2, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:290 +0x191
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*TableName).ResolveExisting(0xc004c9ab40, 0x824efa0, 0xc005b68f60, 0x11180180, 0xc0054833c0, 0x10000, 0xc00561d0a2, 0x3, 0xc005499370, 0x1, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/name_resolution.go:287 +0x791
github.com/cockroachdb/cockroach/pkg/sql.resolveExistingObjectImpl(0x824efa0, 0xc005b68f60, 0x82a34e0, 0xc0054833c0, 0xc004c9ab40, 0x10000, 0x0, 0x10, 0xc005b52760, 0xc0054a00a0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:134 +0x16a
github.com/cockroachdb/cockroach/pkg/sql.ResolveMutableExistingObject(0x824efa0, 0xc005b68f60, 0x82a34e0, 0xc0054833c0, 0xc004c9ab40, 0xc006d3bf00, 0x0, 0x10, 0x75d1960, 0xa3f1d01)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:120 +0x82
github.com/cockroachdb/cockroach/pkg/sql.(*planner).ResolveMutableTableDescriptor(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:195
github.com/cockroachdb/cockroach/pkg/sql.(*planner).prepareDrop(0xc0054833c0, 0x824efa0, 0xc005b68f60, 0xc004c9ab40, 0xc004c9aa00, 0x0, 0x1, 0x1, 0x1)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/drop_table.go:173 +0x75
github.com/cockroachdb/cockroach/pkg/sql.(*planner).DropDatabase(0xc0054833c0, 0x824efa0, 0xc005b68f60, 0xc005874bc0, 0x780bbc0, 0x1, 0xc0062ced70, 0xc006d3c208)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/drop_database.go:101 +0x4ee
github.com/cockroachdb/cockroach/pkg/sql.buildOpaque(0x824efa0, 0xc005b68f60, 0xc0054833d0, 0xc0054834a0, 0x8267ee0, 0xc005874bc0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opaque.go:81 +0x1ac9
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).tryBuildOpaque(0xc0068ce000, 0x8267ee0, 0xc005874bc0, 0xc0061e4800, 0xc0054834a0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/opaque.go:70 +0x131
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt(0xc0068ce000, 0x8267ee0, 0xc005874bc0, 0x0, 0x0, 0x0, 0xc0061e4800, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:322 +0x2f7
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot(0xc0068ce000, 0x8267ee0, 0xc005874bc0, 0x0, 0x0, 0x0, 0xc0061e4800, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:217 +0xe3
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build(0xc0068ce000, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:188 +0x172
github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo(0xc005483ab0, 0x824efa0, 0xc005b68f60, 0x40614d2, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:420 +0x14f
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan(0xc0054833c0, 0x824efa0, 0xc005b68f60, 0x1, 0x2)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:162 +0x7d
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan(0xc005483000, 0x824efa0, 0xc005b68f60, 0xc0054833c0, 0x0, 0x28)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:742 +0x111
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc005483000, 0x824efa0, 0xc005b68f60, 0xc0054833c0, 0x1016e5b8, 0xc000212180, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:624 +0xf9
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc005483000, 0x824efa0, 0xc005b68f60, 0x8267ee0, 0xc005874bc0, 0xc0058b2000, 0x1b, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:412 +0x9aa
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc005483000, 0x824efa0, 0xc005b68f60, 0x8267ee0, 0xc005874bc0, 0xc0058b2000, 0x1b, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:93 +0x4f6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc005483000, 0x824eee0, 0xc0040393c0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1316 +0x1a63
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc005483000, 0x824eee0, 0xc005433c80, 0xc0008cce80, 0x5400, 0x15000, 0xc0008ccf18, 0xc005499310, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1245 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc00085a500, 0x824eee0, 0xc005433c80, 0xc005483000, 0x5400, 0x15000, 0xc0008ccf18, 0xc005499310, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:452 +0x104
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc00548a7ef, 0xc005441fe0, 0x824eee0, 0xc005433c80, 0xc005499310, 0xc00085a500, 0xc000211b00, 0x82668e0, 0xc005441fc0, 0xc00549b020, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:566 +0x2e8
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:498 +0x17e

is conflicting with the information_schema lookup for SHOW TABLES:

goroutine 26656 [select]:
github.com/cockroachdb/cockroach/pkg/storage/txnwait.(*Queue).MaybeWaitForPush(0xc000840600, 0x824efa0, 0xc005dc4960, 0xc0055ac780, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/txnwait/queue.go:506 +0xfe8
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).maybeWaitForPushee(0xc000bf0000, 0x824efa0, 0xc005dc4960, 0xc0052f5900, 0xc005555f20, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_send.go:253 +0xb6
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeBatchWithConcurrencyRetries(0xc000bf0000, 0x824efa0, 0xc005dc4960, 0xc0052f5900, 0x7b6f308, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:185 +0x19d
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc000bf0000, 0x824efa0, 0xc005dc4930, 0x6, 0xc0052f5900, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:95 +0x68a
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc000bf0000, 0x824efa0, 0xc005dc4930, 0x15f2c29c8e95d440, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:37 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000a9d500, 0x824efa0, 0xc005dc48d0, 0x15f2c29c8e95d440, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc000900140, 0x824efa0, 0xc005dc48d0, 0x15f2c29c8e95d440, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x824efa0, 0xc005dc48d0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:935 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0009d0ab0, 0x824efa0, 0xc005dc48d0, 0x79e67ff, 0x10, 0xc0070cccb8, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:301 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000150a00, 0x824efa0, 0xc005dc48d0, 0xc0052f5880, 0xc005dc48d0, 0xc005dc4720, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:923 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000150a00, 0x824efa0, 0xc005dc48a0, 0xc0052f5880, 0xc005dc4840, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:961 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc0053b8a40, 0x824efa0, 0xc005dc48a0, 0x1, 0x8210ba0, 0xc00030f0e0, 0x15f2c29c8e95d440, 0x0, 0x100000001, 0x1, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc0053b8a40, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000318240, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1592 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000318240, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:440 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000318240, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:522 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000318240, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1363 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000318240, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1058 +0x146f
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000318240, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:708 +0x7b4
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0009a3860, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:223 +0x95
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0009a3800, 0x824efa0, 0xc005dc4840, 0x15f2c29c8e95d440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:736
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x824efa0, 0xc005dc4840, 0xc0070ce968, 0xc005af3700, 0xc0070cf050, 0xc0070cebc8)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0009a3800, 0x824efa0, 0xc005dc4840, 0xc005af3700, 0x19, 0xc0070ceb98)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:695 +0x9c
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).MaybePushTransactions(0xc0002ba540, 0x824efa0, 0xc005dc4840, 0xc0070cf050, 0x15f2c29c8e86fb78, 0x0, 0x100000001, 0x1, 0x0, 0x6, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:444 +0x760
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).maybePushIntents(0xc0002ba540, 0x824efa0, 0xc005dc4840, 0xc00473db80, 0x2, 0x2, 0x15f2c29c8e86fb78, 0x0, 0x100000001, 0x1, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:348 +0x279
github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).ProcessWriteIntentError(0xc0002ba540, 0x824efa0, 0xc005dc4840, 0xc006bfb4a0, 0x15f2c29c8e86fb78, 0x0, 0x100000001, 0x1, 0x0, 0x6, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:265 +0x147
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleWriteIntentError(0xc000bf0000, 0x824efa0, 0xc005dc4840, 0xc0052f5780, 0xc006bfb4a0, 0xc005ddce00, 0x0, 0xc006bfb4a0, 0xc0070cf910)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:281 +0x14b
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeBatchWithConcurrencyRetries(0xc000bf0000, 0x824efa0, 0xc005dc4840, 0xc0052f5780, 0x7b6f2e8, 0x0, 0xc006bfb4a0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:205 +0x40f
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc000bf0000, 0x824efa0, 0xc005dc4810, 0x6, 0xc0052f5780, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:99 +0x2f3
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc000bf0000, 0x824efa0, 0xc005dc4810, 0x15f2c29c8e86fb78, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_send.go:37 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000a9d500, 0x824efa0, 0xc005dc47b0, 0x15f2c29c8e86fb78, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc000900140, 0x824efa0, 0xc005dc47b0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x824efa0, 0xc005dc47b0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:935 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0009d0ab0, 0x824efa0, 0xc005dc47b0, 0x79e67ff, 0x10, 0xc0070d05c8, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:301 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000150a00, 0x824efa0, 0xc005dc47b0, 0xc0052f5700, 0xc005dc47b0, 0x80, 0x80)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:923 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000150a00, 0x824efa0, 0xc005dc4780, 0xc0052f5700, 0xc005dc4720, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:961 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc0053b8840, 0x824efa0, 0xc005dc4780, 0x1, 0x8210ba0, 0xc00030f0e0, 0x0, 0x0, 0x100000001, 0x1, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc0053b8840, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000318240, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1592 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000318240, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:440 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000318240, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:522 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000318240, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1363 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000318240, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1058 +0x146f
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000318240, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:708 +0x7b4
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc006348888, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetricRecorder).SendLocked(0xc006348850, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metric_recorder.go:46 +0x8d
github.com/cockroachdb/cockroach/pkg/kv.(*txnCommitter).SendLocked(0xc006348820, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_committer.go:126 +0x6f9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendHelper(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:345
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc0063487b8, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:230 +0xa9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc0063487b8, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:179 +0x158
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc006348708, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:208 +0x159
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc0063486e8, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).SendLocked(0xc006348648, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:168 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc006348480, 0x824efa0, 0xc005dc4720, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:488 +0x406
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0009a3800, 0x824eee0, 0xc0053b8600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc00633b680, 0x824eee0, 0xc0053b8600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:870 +0x11e
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x824eee0, 0xc0053b8600, 0xc0070d2f48, 0xc005af3180, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc00633b680, 0x824eee0, 0xc0053b8600, 0xc005af3180, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:538 +0xfa
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).scan(0xc00633b680, 0x824eee0, 0xc0053b8600, 0x774cc80, 0xc005ddcce0, 0x774cc80, 0xc005ddcd00, 0x0, 0xc005ddcc00, 0x79c8856, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:445 +0xfd
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Scan(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:459
github.com/cockroachdb/cockroach/pkg/sql.GetAllDescriptors(0x824eee0, 0xc0053b8600, 0xc00633b680, 0x203001, 0x203001, 0x203001, 0xc0070d3298, 0x404843c)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/descriptor.go:300 +0x167
github.com/cockroachdb/cockroach/pkg/sql.(*TableCollection).getAllDescriptors(0xc005b2f238, 0x824eee0, 0xc0053b8600, 0xc00633b680, 0x50, 0x48, 0x7712320, 0xc00436ede0, 0xc0070d32f8)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:721 +0x8e
github.com/cockroachdb/cockroach/pkg/sql.forEachTableDescWithTableLookupInternal(0x824eee0, 0xc0053b8600, 0xc005b2f3c0, 0xc005ddcc80, 0x0, 0x0, 0xc0070d33c8, 0xc006bfb450, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/information_schema.go:1620 +0x69
github.com/cockroachdb/cockroach/pkg/sql.forEachTableDescWithTableLookup(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/information_schema.go:1581
github.com/cockroachdb/cockroach/pkg/sql.forEachTableDesc(0x824eee0, 0xc0053b8600, 0xc005b2f3c0, 0xc005ddcc80, 0x0, 0xc0070d3428, 0x50, 0x76edca0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/information_schema.go:1510 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.glob..func78(0x824eee0, 0xc0053b8600, 0xc005b2f3c0, 0xc005ddcc80, 0xc006bfb450, 0xc006bfb400, 0x1)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/information_schema.go:1358 +0x81
github.com/cockroachdb/cockroach/pkg/sql.virtualDefEntry.getPlanInfo.func1(0x824eee0, 0xc0053b8600, 0xc005b2f3c0, 0xc000bad21e, 0x9, 0xc00004e000, 0x74fd440, 0x75a0ae0, 0x1108b188)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/virtual_schema.go:274 +0x354
github.com/cockroachdb/cockroach/pkg/sql.(*execFactory).ConstructVirtualScan.func1(0x824eee0, 0xc0053b8600, 0xc005b2f3c0, 0x400cafa, 0x74fd440, 0x75a0ae0, 0xc00568fe01)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_exec_factory.go:145 +0x5d
github.com/cockroachdb/cockroach/pkg/sql.(*delayedNode).startExec(0xc0053b8480, 0x824eee0, 0xc0053b8600, 0xc006105900, 0xc005b2f3c0, 0xc006270600, 0xc00633b680)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/delayed.go:48 +0x57
github.com/cockroachdb/cockroach/pkg/sql.startExec.func2(0x79df2f8, 0xd, 0x82507a0, 0xc0053b8480, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:376 +0xb9
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1(0xc0070d3d28, 0x79df2f8, 0xd, 0x82507a0, 0xc0053b8480)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:143 +0x5d
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc0070d3d28, 0x82507a0, 0xc0053b8480, 0x79df2f8, 0xd)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:708 +0x2a6
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc0070d3d28, 0x82507a0, 0xc0053b8480, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:110 +0x8d
github.com/cockroachdb/cockroach/pkg/sql.walkPlan(...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:74
github.com/cockroachdb/cockroach/pkg/sql.startExec(0x824eee0, 0xc0053b8600, 0xc006105900, 0xc005b2f3c0, 0x82507a0, 0xc0053b8480, 0x0, 0xc003e35052)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:379 +0x1bb
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start(0xc005bd4500, 0x824eee0, 0xc0053b8600, 0xc003e35015, 0xf)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:117 +0xd6
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*sortAllProcessor).Start(0xc005bd4a00, 0x824eee0, 0xc0053b8600, 0xa47abe0, 0x73a6ca0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/sorter.go:274 +0x52
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).Run(0xc005bd4a00, 0x824eee0, 0xc0053b8600)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:748 +0x52
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc0062706c0, 0x824eee0, 0xc0053b8600, 0x7b6cec0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:370 +0x247
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc00014e640, 0xc005555bc0, 0xc00633b680, 0xc0070d4558, 0xc003b82840, 0xc005b2f4a0, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:377 +0x426
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc00014e640, 0x824efa0, 0xc005691e30, 0xc005b2f4a0, 0xc005555bc0, 0xc00633b680, 0x8251560, 0xc005dc4510, 0xc003b82840, 0xc005b2f1d8)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:987 +0x225
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc005b2f000, 0x824efa0, 0xc005691e30, 0xc005b2f3c0, 0x3, 0x1016e5b8, 0xc006432680, 0xc0070d4900, 0xc006bfaef8, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:826 +0x3cf
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc005b2f000, 0x824efa0, 0xc005691e30, 0xc005b2f3c0, 0x1016e5b8, 0xc006432680, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:716 +0x70a
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc005b2f000, 0x824efa0, 0xc005691e30, 0x8269220, 0xc005691c50, 0xc003e38000, 0xb, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:412 +0x9aa
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc005b2f000, 0x824efa0, 0xc005691e30, 0x8269220, 0xc005691c50, 0xc003e38000, 0xb, 0x0, 0x0, 0x0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:93 +0x4f6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc005b2f000, 0x824eee0, 0xc004e9bd40, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1316 +0x1a63
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc005b2f000, 0x824eee0, 0xc004e9bd00, 0xc0008cce80, 0x5400, 0x15000, 0xc0008ccf18, 0xc005b5ac50, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1245 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc00085a500, 0x824eee0, 0xc004e9bd00, 0xc005b2f000, 0x5400, 0x15000, 0xc0008ccf18, 0xc005b5ac50, 0x0, 0x0)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:452 +0x104
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc006128bee, 0xc005ddc3e0, 0x824eee0, 0xc004e9bd00, 0xc005b5ac50, 0xc00085a500, 0xc006432000, 0x82668e0, 0xc005ddc3c0, 0xc0055556e0, ...)
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:566 +0x2e8
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	/Users/otan/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:498 +0x17e

@jordanlewis
Copy link
Member

@ajwerner any thoughts on this?

@otan
Copy link
Contributor

otan commented Feb 12, 2020 •
edited
Loading

also scarily after this happens, after restarting ./cockroach start-single-node, DROP DATABASE IF EXISTS db0;CREATE DATABASE db0;USE db0;CREATE TABLE t0(c0 INT);' does not succeed afterwards and I need to drop my internal stores to get this to work again,

@ajwerner
Copy link
Contributor

Well one thing to note is that this is pretty new. I can't repro on 19.2.3 but it seems to repro instantly on master.

@ajwerner
Copy link
Contributor

Well this one is super duper fun! and indicative of problems lurking in the deep. The problem seems to be that we have a DROP DATABASE command which hits a TransactionRetryWithProtoRefreshError. The connExecutor is happy to internally rerun this transaction in a new epoch but the intents will remain. Our schema accessors underneath use their own transactions. When that new transaction encounters the intent from the previous epoch of the current transaction it blocks, hence the deadlock.

My one remaining question is whether the interaction with caches has changed in such a way that this case is now much more common.

The recommended solution is to force the connExecutor to abandon the current transaction and cleanup its intents when retrying a transaction which performs any DDLs.

For now let's fold these deadlocks into the below issue:
#24885

@ajwerner ajwerner mentioned this issue Feb 12, 2020
Closed
@otan
Copy link
Contributor

otan commented Mar 23, 2020

cc @mrigger this should be fully resolved after #46384!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thoszhang thoszhang

Labels
A-schema-descriptors Relating to SQL table/db descriptor handling. A-sql-vtables Virtual tables - pg_catalog, information_schema etc C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@jordanlewis @knz @solongordon @ajwerner @thoszhang @otan @yuzefovich @mrigger