fix trie by asking peers #4312
Merged
fix trie by asking peers #4312
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
|
Hi folks, I'm using Besu v22.7.1 with Nimbus v22.8.1 on Goerli/Prater and ran into an issue that @garyschulte recommended I provide here, so you can gauge whether or not this PR would fix it. Yesterday my Besu node ran into a critical error and it stopped importing new blocks: At which point, Nimbus started printing the following over and over again and stopped syncing: This was on one of my Does this look like it would be caught by this PR and resolved autonomously, or is there something else to do here? |
do you still have the error? if so could you run this PR to see if it fixes the problem ? |
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
jflo
requested changes
Sep 6, 2022
| return maybeFallbackNodeFinder; | ||
| } | ||
|
|
||
| public void addFallbackNodeFinder(final Optional<PeerTrieNodeFinder> maybeFallbackNodeFinder) { |
There was a problem hiding this comment.
When I saw this was called addFallbackNodeFinder, I expected it to be adding to a collection. This is not the case as there is only a single member to use for fallback. perhaps setFallbackNodeFiner or useFallbackNodeFinder would be more intuitive.
...ain/java/org/hyperledger/besu/ethereum/eth/sync/worldstate/WorldStatePeerTrieNodeFinder.java
Outdated
Show resolved
Hide resolved
...ain/java/org/hyperledger/besu/ethereum/eth/sync/worldstate/WorldStatePeerTrieNodeFinder.java
Outdated
Show resolved
Hide resolved
| msg.capability, new DefaultMessage(peerConnection, response)); | ||
| } else | ||
| snapProtocolManager.ifPresent( | ||
| protocolManager -> |
There was a problem hiding this comment.
Could also wrap this in a capability check to make sure snap manager can handle it.
There was a problem hiding this comment.
not sure to understand this one ? modifying this test class or another part ?
There was a problem hiding this comment.
For this test class. It will exercise the capabilities collection checking.
| if (response.containsKey(nodeHash)) { | ||
| LOG.trace("Found node {} with getNodeData request", nodeHash); | ||
| return Optional.of(response.get(nodeHash)); | ||
| } |
There was a problem hiding this comment.
else case isn't handled, I think it might warrant a TRACE statement just like the if case.
| if (nodeValue != null && Hash.hash(nodeValue).equals(nodeHash)) { | ||
| LOG.trace("Found node {} with getTrieNode request", nodeHash); | ||
| return Optional.of(nodeValue); | ||
| } |
There was a problem hiding this comment.
same suggestion here, to better track our assumptions.
ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/DefaultSynchronizer.java
Show resolved
Hide resolved
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
jflo
reviewed
Sep 6, 2022
| @@ -101,6 +101,8 @@ private DefaultBlockchain( | |||
| chainHeadTransactionCount = chainHeadBody.getTransactions().size(); | |||
| chainHeadOmmerCount = chainHeadBody.getOmmers().size(); | |||
|
|
|||
| System.out.println("head " + getChainHead().getHeight() + " " + getChainHead().getHash()); | |||
There was a problem hiding this comment.
probably meant to use a Logger or remove?
There was a problem hiding this comment.
for test forgot to remove it
jflo
approved these changes
Sep 6, 2022
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Yes, I've actually had pretty consistent failures on Prater using 22.7.1 (and later a 22.7.2 snapshot from a few days ago). It's always related to a Here's another example: I will build an image with this PR and try it out. |
|
Unfortunately, this PR didn't seem to fix my problem - it seems to have made it considerably worse; I can barely keep Besu synced for an hour before it breaks with constant log messages such as: It looks like RocksDB is constantly timing out. Perhaps there's a setting somewhere that is just overly aggressively tuned? CC @garyschulte, did that change between 22.7.1 and this PR? I've attached a full log file here for reference so you can see the stream of errors... hopefully there is some insight to be gleaned: |
garyschulte
approved these changes
Sep 7, 2022
garyschulte
pushed a commit
to garyschulte/besu
that referenced
this pull request
Sep 7, 2022
* fix trie by asking peers Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
garyschulte
pushed a commit
to garyschulte/besu
that referenced
this pull request
Sep 7, 2022
* fix trie by asking peers Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
garyschulte
pushed a commit
to garyschulte/besu
that referenced
this pull request
Sep 7, 2022
* fix trie by asking peers Signed-off-by: Karim TAAM <karim.t2am@gmail.com> Signed-off-by: garyschulte <garyschulte@gmail.com>
eum602
pushed a commit
to lacchain/besu
that referenced
this pull request
Nov 3, 2023
* fix trie by asking peers Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Karim TAAM karim.t2am@gmail.com
PR description
we may have an inconsistency in the worldstate trie. If this happens even if the cases are very rare and complex to reproduce this PR can help resolve the issue. Indeed if we detect a problem in the trie we can fix it directly by asking our peers for the correct version of the node
We start by asking with the GetNodeData request (which will soon be deprecated but which is still compatible and more practical) and if the response is empty we try with GetTrieNode
Fixed Issue(s)
Documentation
doc-change-requiredlabel to this PR ifupdates are required.
Changelog