Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unclear error message for failed analyze jobs #52071

Closed
aytrack opened this issue Mar 25, 2024 · 1 comment
Closed

unclear error message for failed analyze jobs #52071

aytrack opened this issue Mar 25, 2024 · 1 comment
Assignees
Labels
severity/moderate sig/planner SIG: Planner type/bug The issue is confirmed as a bug.

Comments

@aytrack
Copy link
Contributor

aytrack commented Mar 25, 2024

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. analyze a table and make sure the analyze job will be killed by global control

2. What did you expect to see? (Required)

should return the analyze jobs was killed by global control

MySQL [rtdb]> analyze table xxx with 1 samplerate;
Error: Your query has been cancelled due to exceeding the allowed memory limit for the tidb-server instance and this query is currently using the most memory. Please try narrowing your query scope or increase the tidb_server_memory_limit and try again

3. What did you see instead (Required)

MySQL [rtdb]> analyze table xxx with 1 samplerate;
ERROR 1105 (HY000): runtime error: invalid memory address or nil pointer dereference
[2024/03/25 17:03:35.351 +08:00] [WARN] [servermemorylimit.go:149] ["global memory controller tries to kill the top1 memory consumer"] [conn=2673868862] ["sql digest"=b23bbd59f5f16560b4ca58e7eb13cffb054634264d9d0e23a4af0a38110b9c16] ["sql text"="analyze table xxx with 1 samplerate"] [tidb_server_memory_limit=3221225472] ["heap inuse"=3249176576] ["sql memory usage"=1788292799]
[2024/03/25 17:03:35.946 +08:00] [INFO] [backoff.go:389] ["backoff stops because a killed signal is received"] [signal=4]
[2024/03/25 17:03:35.947 +08:00] [INFO] [coprocessor.go:1927] ["a killed signal is received in copIteratorWorker"] [signal=4]
[2024/03/25 17:03:36.331 +08:00] [INFO] [backoff.go:389] ["backoff stops because a killed signal is received"] [signal=4]
[2024/03/25 17:03:36.331 +08:00] [INFO] [coprocessor.go:1927] ["a killed signal is received in copIteratorWorker"] [signal=4]
[2024/03/25 17:03:36.506 +08:00] [INFO] [backoff.go:389] ["backoff stops because a killed signal is received"] [signal=4]
[2024/03/25 17:03:36.506 +08:00] [INFO] [coprocessor.go:1927] ["a killed signal is received in copIteratorWorker"] [signal=4]
[2024/03/25 17:03:36.681 +08:00] [INFO] [backoff.go:389] ["backoff stops because a killed signal is received"] [signal=4]
[2024/03/25 17:03:36.681 +08:00] [INFO] [coprocessor.go:1927] ["a killed signal is received in copIteratorWorker"] [signal=4]
[2024/03/25 17:03:36.811 +08:00] [WARN] [sqlkiller.go:72] ["global memory controller, NeedKill signal is received successfully"] [conn=2673868862]
[2024/03/25 17:03:36.825 +08:00] [ERROR] [analyze_col_v2.go:638] ["analyze worker panicked"] [recover="[executor:8176]Your query has been cancelled due to exceeding the allowed memory limit for the tidb-server instance and this query is currently using the most memory. Please try narrowing your query scope or increase the tidb_server_memory_limit and try again.[conn=2673868862]"] [stack="github.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).subMergeWorker.func1\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:638\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\ngithub.com/pingcap/tidb/pkg/util/memory.(*Tracker).Consume\n\t/workspace/source/tidb/pkg/util/memory/tracker.go:459\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).subMergeWorker\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:685\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).buildSamplingStats.func3\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:297\ngithub.com/tiancaiamao/gp.workerLoop\n\t/root/go/pkg/mod/github.com/tiancaiamao/gp@v0.0.0-20221230034425-4025bc8a4d4a/gp.go:72\ngithub.com/tiancaiamao/gp.worker\n\t/root/go/pkg/mod/github.com/tiancaiamao/gp@v0.0.0-20221230034425-4025bc8a4d4a/gp.go:51"]
[2024/03/25 17:03:37.198 +08:00] [WARN] [sqlkiller.go:72] ["global memory controller, NeedKill signal is received successfully"] [conn=2673868862]
[2024/03/25 17:03:37.198 +08:00] [ERROR] [analyze_col_v2.go:638] ["analyze worker panicked"] [recover="[executor:8176]Your query has been cancelled due to exceeding the allowed memory limit for the tidb-server instance and this query is currently using the most memory. Please try narrowing your query scope or increase the tidb_server_memory_limit and try again.[conn=2673868862]"] [stack="github.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).subMergeWorker.func1\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:638\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\ngithub.com/pingcap/tidb/pkg/util/memory.(*Tracker).Consume\n\t/workspace/source/tidb/pkg/util/memory/tracker.go:459\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).subMergeWorker\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:685\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).buildSamplingStats.func3\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:297\ngithub.com/tiancaiamao/gp.workerLoop\n\t/root/go/pkg/mod/github.com/tiancaiamao/gp@v0.0.0-20221230034425-4025bc8a4d4a/gp.go:72\ngithub.com/tiancaiamao/gp.worker\n\t/root/go/pkg/mod/github.com/tiancaiamao/gp@v0.0.0-20221230034425-4025bc8a4d4a/gp.go:51"]
[2024/03/25 17:03:37.198 +08:00] [ERROR] [analyze.go:520] ["analyze worker panicked"] [recover="runtime error: invalid memory address or nil pointer dereference"] [stack="github.com/pingcap/tidb/pkg/executor.(*AnalyzeExec).analyzeWorker.func1\n\t/workspace/source/tidb/pkg/executor/analyze.go:520\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:914\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:261\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:861\ngithub.com/pingcap/tidb/pkg/distsql.(*selectResult).Close\n\t/workspace/source/tidb/pkg/distsql/select_result.go:610\ngithub.com/pingcap/tidb/pkg/executor.closeAll\n\t/workspace/source/tidb/pkg/executor/distsql.go:138\ngithub.com/pingcap/tidb/pkg/executor.(*tableResultHandler).Close\n\t/workspace/source/tidb/pkg/executor/table_reader.go:623\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).buildSamplingStats.func1\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:266\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).buildSamplingStats\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:340\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).analyzeColumnsPushDownV2\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:149\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeColumnsExecV2).analyzeColumnsPushDownWithRetryV2\n\t/workspace/source/tidb/pkg/executor/analyze_col_v2.go:58\ngithub.com/pingcap/tidb/pkg/executor.analyzeColumnsPushDownEntry\n\t/workspace/source/tidb/pkg/executor/analyze_col.go:67\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeExec).analyzeWorker\n\t/workspace/source/tidb/pkg/executor/analyze.go:541\ngithub.com/pingcap/tidb/pkg/executor.(*AnalyzeExec).Next.func1\n\t/workspace/source/tidb/pkg/executor/analyze.go:119\ngithub.com/pingcap/tidb/pkg/util.(*WaitGroupPool).Run.func1\n\t/workspace/source/tidb/pkg/util/wait_group_wrapper.go:213\ngithub.com/tiancaiamao/gp.workerLoop\n\t/root/go/pkg/mod/github.com/tiancaiamao/gp@v0.0.0-20221230034425-4025bc8a4d4a/gp.go:72\ngithub.com/tiancaiamao/gp.worker\n\t/root/go/pkg/mod/github.com/tiancaiamao/gp@v0.0.0-20221230034425-4025bc8a4d4a/gp.go:51"]
[2024/03/25 17:03:37.198 +08:00] [ERROR] [analyze.go:423] ["analyze failed"] [conn=2673868862] [session_alias=] [error="runtime error: invalid memory address or nil pointer dereference"]

4. What is your TiDB version? (Required)

MySQL [rtdb]> select tidb_version();
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                  |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.0.0-alpha-619-g90b21d9
Edition: Community
Git Commit Hash: 90b21d95b865c3910c26cfaba689b70d345f87a8
Git Branch: HEAD
UTC Build Time: 2024-03-25 07:59:25
GoVersion: go1.21.4
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
@Rustin170506
Copy link
Member

same root cause as this #51928, Introduced by #51787.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
severity/moderate sig/planner SIG: Planner type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

2 participants