-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
1105 runtime error: invalid memory address or nil pointer dereference #19041
Comments
/label component/executor |
|
@cberescu could you please give more information on this issue? e.g. how do you find it and how it will likely to occur? |
It is hard to duplicate it, it appears in a complex query, i will attach the query if you think it may help here :
If it is anything else i can give or require please let me know i am available to help in any way i can. |
@cberescu thank you for being so responsive, I'll be looking into this issue, more over, could you please a more specific version number? Say, the output of |
Hi, the version was 4.0.0 , i have now updated to 4.0.3 , I can give a try and see it this happens with this version also. |
Cool, thanks a lot. By the way, how frequent will this error pops up? |
I can not tell you exactly how often it pops up, it did 2 times pretty close to each other within an hour, after that i moved the query back to mysql. At this moment it is running for about 3 hours and nothing has happened. I will let it run for the next 24 or 48 hours and if nothing happens then i think we can conclude the issue was fixed with the 4.0.3 version . I will keep you informed if anything happens or it doesn't. Thank you. |
Cool, thank you so much for your warm heart, let's work it out together! Looking forward to your update! |
It looks like a data race when BatchPointGet is executed in multi-thread context. You can enable |
Hey guys, I have let the query where i found the issue to be processed by tidb for the last 48h and i did not see for the error to appear again. So it seems maybe there was a fix between version 4.0.0 and 4.0.3 I will anyway keep it running like this for the time being. |
Could you try @eurekaka 's suggestion, by rebuilding the tidb-server with |
Hey @ichn-hu unfortunately i do not know how to do that, i used the tiup to deploy the cluster. Is this something i can do with it? If you have a documentation about how i can do it i can deploy a instance with the flag set and send all traffic to it. |
Hi @cberescu , sorry for a late reply, basically you need to compile & build the tidb server by yourself with go's race detector turned on. To compile the tidb-server, you need to have a go environment, then clone the code repository, and use
to compile the tidb server with the race detector. And you could then test against the compiled binary file, which will detect race condition on the fly. Let me know if you have any more questions. |
More on this, you might not need to use tiup to test the binary file in a cluster, you could just If you need to use tiup in any case, if you are using |
for now i do not seem to receive the error anymore, could it have been fixed maybe in one of releases since 4.0.0? As i said the error was received on the 4.0.0 version. |
Probably. Perhaps we could consider close this issue? @cberescu feel free to reopen it as long as the problem accur again, thanks for your patience and help~ |
@SunRunAway would you please help close this issue? |
After investigation, this issue is confirmed to be resolved by later PRs, namely #18145, and the cause is identified here: #16817 (comment) . In short, the BatchPointGetExec tries to get a snapshot from the txnState, which after checking validity, might be concurrently changed before @cberescu as long as you are using later release versions, such as 4.0.5, you are now free from such panic. |
that is great news, thanks for your explanation and patience with me. |
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
Can not really reproduce, it does not happen all the time.
2. What did you expect to see? (Required)
To not receive a error
3. What did you see instead (Required)
General error: 1105 runtime error: invalid memory address or nil pointer dereference
[2020/08/06 11:11:05.744 +02:00] [ERROR] [misc.go:88] ["panic in the recoverable goroutine"] [r=""invalid memory address or nil pointer dereference""] ["stack trace"="github.com/pingcap/tidb/util.WithRecovery.func1\n\t/home/jenkins/age
nt/workspace/tidb_v4.0.0/go/src/github.com/pingcap/tidb/util/misc.go:90\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:679\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:199\nruntime.sigpanic\n\t/usr/local/go/src/runtime
/signal_unix.go:394\ngithub.com/pingcap/tidb/session.(*TxnState).GetSnapshot\n\t:1\ngithub.com/pingcap/tidb/executor.(*BatchPointGetExec).initialize\n\t/home/jenkins/agent/workspace/tidb_v4.0.0/go/src/github.com/pingcap/ti
db/executor/batch_point_get.go:128\ngithub.com/pingcap/tidb/executor.(*BatchPointGetExec).Next\n\t/home/jenkins/agent/workspace/tidb_v4.0.0/go/src/github.com/pingcap/tidb/executor/batch_point_get.go:89\ngithub.com/pingcap/tidb/executor.N
ext\n\t/home/jenkins/agent/workspace/tidb_v4.0.0/go/src/github.com/pingcap/tidb/executor/executor.go:248\ngithub.com/pingcap/tidb/executor.(*HashJoinExec).fetchBuildSideRows\n\t/home/jenkins/agent/workspace/tidb_v4.0.0/go/src/github.com/
pingcap/tidb/executor/join.go:257\ngithub.com/pingcap/tidb/executor.(*HashJoinExec).fetchAndBuildHashTable.func1\n\t/home/jenkins/agent/workspace/tidb_v4.0.0/go/src/github.com/pingcap/tidb/executor/join.go:675\ngithub.com/pingcap/tidb/ut
il.WithRecovery\n\t/home/jenkins/agent/workspace/tidb_v4.0.0/go/src/github.com/pingcap/tidb/util/misc.go:93"]
4. Affected version (Required)
v 4.00
5. Root Cause Analysis
The text was updated successfully, but these errors were encountered: