Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IndexHashJoin hang when exceed tidb_mem_quota_query #49033

Closed
wshwsh12 opened this issue Nov 30, 2023 · 1 comment · Fixed by #49218
Closed

IndexHashJoin hang when exceed tidb_mem_quota_query #49033

wshwsh12 opened this issue Nov 30, 2023 · 1 comment · Fixed by #49218
Labels
affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 affects-7.5 severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.

Comments

@wshwsh12
Copy link
Contributor

wshwsh12 commented Nov 30, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

use test;
drop table if exists t, s;
create table t(a int, index(a));
create table s(a int, index(a));
insert into t values(1), (2), (3), (4), (5), (6), (7), (8), (9), (10), (11), (12), (13), (14), (15), (16), (17), (18), (19), (20), (21), (22), (23), (24), (25), (26), (27), (28), (29), (30), (31), (32), (33), (34), (35), (36), (37), (38), (39), (40), (41), (42), (43), (44), (45), (46), (47), (48), (49), (50), (51), (52), (53), (54), (55), (56), (57), (58), (59), (60), (61), (62), (63), (64), (65), (66), (67), (68), (69), (70), (71), (72), (73), (74), (75), (76), (77), (78), (79), (80), (81), (82), (83), (84), (85), (86), (87), (88), (89), (90), (91), (92), (93), (94), (95), (96), (97), (98), (99), (100), (101), (102), (103), (104), (105), (106), (107), (108), (109), (110), (111), (112), (113), (114), (115), (116), (117), (118), (119), (120), (121), (122), (123), (124), (125), (126), (127), (128);
insert into s values(1), (128);
set @@tidb_max_chunk_size=32;
set @@tidb_index_lookup_join_concurrency=1;
set @@tidb_index_join_batch_size=32;
desc format = 'brief' select /*+ INL_HASH_JOIN(s) */ * from t join s on t.a=s.a order by t.a;
set @@tidb_mem_quota_query=12800;
select /*+ INL_HASH_JOIN(s) */ * from t join s on t.a=s.a order by t.a;

(In v6.1, v5.4,using tidb_mem_quota_query = 5000)

2. What did you expect to see? (Required)

Can cancel the sql or run successfully.

3. What did you see instead (Required)

Hang forever.

goroutine 18480 [select, 5 minutes]:
github.com/pingcap/tidb/pkg/executor.(*IndexNestedLoopHashJoin).runInOrder(0xc0012c2600, {0x6365340, 0xc002dfc510}, 0xc002e214f0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/index_lookup_hash_join.go:260 +0xdd
github.com/pingcap/tidb/pkg/executor.(*IndexNestedLoopHashJoin).Next(0x6365340?, {0x6365340, 0xc002dfc510}, 0xc002e214f0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/index_lookup_hash_join.go:225 +0x259
github.com/pingcap/tidb/pkg/executor/internal/exec.Next({0x6365340, 0xc002dfc510}, {0x637ab88, 0xc0012c2600}, 0xc002e214f0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:289 +0x2ba
github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next(0xc001042d20, {0x6365340, 0xc002dfc510}, {0x637ab88, 0xc0012c2600}, 0xc0019cc400?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:1226 +0x6e
github.com/pingcap/tidb/pkg/executor.(*recordSet).Next(0xc002e214a0, {0x6365340?, 0xc002dfc510?}, 0xc002e214f0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:154 +0xb2
github.com/pingcap/tidb/pkg/server/internal/resultset.(*tidbResultSet).Next(0xc0019cc000?, {0x6365340?, 0xc002dfc510?}, 0xc0034f6980?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/internal/resultset/resultset.go:62 +0x25
github.com/pingcap/tidb/pkg/server.(*clientConn).writeChunks(0xc002817500, {0x6365340, 0xc002dfc510}, {0x6374da0, 0xc0015e18c0}, 0x0, 0x1bd?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2240 +0x18a
github.com/pingcap/tidb/pkg/server.(*clientConn).writeResultSet(0xc002817500, {0x6365340, 0xc002dfc510}, {0x6374da0, 0xc0015e18c0}, 0x60?, 0x2, 0x0?)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2183 +0x2f0
github.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt(0xc002817500, {0x6365378, 0xc003a1eaf0}, {0x6379ae8, 0xc00573e7e0}, {0x0, 0x0, 0x0}, 0x1)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2051 +0x3e5
github.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery(0xc002817500, {0x6365378, 0xc003a1eaf0}, {0xc00e8323c1, 0x46})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1812 +0x9a5
github.com/pingcap/tidb/pkg/server.(*clientConn).dispatch(0xc002817500, {0x6365340?, 0xc0038a5a70?}, {0xc00e8323c0, 0x47, 0x47})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1299 +0xf8b
github.com/pingcap/tidb/pkg/server.(*clientConn).Run(0xc002817500, {0x6365340, 0xc0038a5a70})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1078 +0x53e
github.com/pingcap/tidb/pkg/server.(*Server).onConn(0xc004383b00?, 0xc002817500)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:701 +0x89d
created by github.com/pingcap/tidb/pkg/server.(*Server).startNetworkListener in goroutine 16348
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:517 +0x78d

4. What is your TiDB version? (Required)

master, v7.5, v7.1, v6.5, v6.1,v5.4

@XuHuaiyu
Copy link
Contributor

XuHuaiyu commented Dec 4, 2023

[2023/12/04 15:59:22.829 +08:00] [ERROR] [misc.go:91] ["panic in the recoverable goroutine"] [r="[executor:8175]Your query has been cancelled due to exceeding the all
owed memory limit for a single SQL query. Please try narrowing your query scope or increase the tidb_mem_quota_query limit and try again.[conn=2950692874]"] ["stack trace"="github.com/pingcap/tidb/pkg/util.WithRecovery.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/misc.go:93
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:914
github.com/pingcap/tidb/pkg/util/memory.(*PanicOnExceed).Action
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/memory/action.go:173
github.com/pingcap/tidb/pkg/util/memory.(*Tracker).Consume.func1
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/memory/tracker.go:437
github.com/pingcap/tidb/pkg/util/memory.(*Tracker).Consume
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/memory/tracker.go:464
github.com/pingcap/tidb/pkg/executor.buildKvRangesForIndexJoin
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/builder.go:4686
github.com/pingcap/tidb/pkg/executor.(*dataReaderBuilder).buildIndexReaderForIndexJoin
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/builder.go:4419
github.com/pingcap/tidb/pkg/executor.(*dataReaderBuilder).buildExecutorForIndexJoinInternal
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/builder.go:4093
github.com/pingcap/tidb/pkg/executor.(*dataReaderBuilder).buildExecutorForIndexJoin
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/builder.go:4084
github.com/pingcap/tidb/pkg/executor.(*innerWorker).fetchInnerResults
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/index_lookup_join.go:698
github.com/pingcap/tidb/pkg/executor.(*indexHashJoinInnerWorker).fetchInnerResults
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/index_lookup_hash_join.go:603
github.com/pingcap/tidb/pkg/executor.(*indexHashJoinInnerWorker).handleTask
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/index_lookup_hash_join.go:659
github.com/pingcap/tidb/pkg/executor.(*indexHashJoinInnerWorker).run
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/index_lookup_hash_join.go:497
github.com/pingcap/tidb/pkg/executor.(*IndexNestedLoopHashJoin).startWorkers.func2
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/index_lookup_hash_join.go:182
github.com/pingcap/tidb/pkg/util.WithRecovery
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/util/misc.go:96"]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 affects-7.5 severity/major sig/execution SIG execution type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants