add retry logic on sync pipeline for rocksdb issue

[matkt]

PR description

This Pull Request allows retrying recoverable RocksDB errors such as 'busy' or 'lock timeout' during the initial synchronization. This helps to unblock the pipeline if possible and prevents pipelines from getting stuck at 99%.

Fixed Issue(s)

#5911

[github-actions]

• I thought about documentation and added the doc-change-required label to this PR if updates are required.
• I thought about the changelog and included a changelog update if required.
• If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

[garyschulte]

Looks good, I will try on the node I have been abusing. Some comments

[garyschulte]

less brittle to use enum values

Suggested change

	return e.getMessage().contains(rocksdbClassName)
	&& (e.getMessage().contains(ERR_BUSY) || e.getMessage().contains(ERR_LOCK_TIMED_OUT));
	}
	return Optional.of(e.getCause())
	.filter(z -> z instanceof RocksDBException)
	.map(RocksDBException.class::cast)
	.map(RocksDBException::getStatus)
	.map(Status::getCode)
	.map(RETRYABLE_STATUS_CODES::contains)
	.orElse(false);

[matkt]

should be good, will update

[garyschulte]

can we not fail the future for an individual task in the foreach rather than failing the whole list? or is that a lot more plumbing to do in the pipeline to handle a single failed task future?

[matkt]

not possible because all modification are in the same updater and we cannot commit only a part of a transaction

[garyschulte]

🙏

[garyschulte]

FYI I am getting persistent and repeated timeouts with this PR when snapsyncing forest. The retry might be grouping together requests that are getting overlapping range responses. I will let it run for a few more hours and see if it makes any headway or if it is deadlocked.

edit: the deadlock did resolve, but it took a couple hours. Once resolved, sync progressed normally until it hit another problematic range. The test instance I used today spent ~6 hours failing and retrying every second, after 12 hours it is roughly 37% complete. So, 👍 it does progress, but it seems like there should be a better / faster strategy we can use.

[matkt]

FYI I am getting persistent and repeated timeouts with this PR when snapsyncing forest. The retry might be grouping together requests that are getting overlapping range responses. I will let it run for a few more hours and see if it makes any headway or if it is deadlocked.

edit: the deadlock did resolve, but it took a couple hours. Once resolved, sync progressed normally until it hit another problematic range. The test instance I used today spent ~6 hours failing and retrying every second, after 12 hours it is roughly 37% complete. So, 👍 it does progress, but it seems like there should be a better / faster strategy we can use.

yes clearly I think maybe we need to take some times to avoid writing same datas in differents thread but could be a mitigation . I also think verkle trie sync will simplify the snapsync stragegy as we have only one trie so maybe waiting 🤔

[matkt]

I restarted a sync to test

[matkt]

FYI I am getting persistent and repeated timeouts with this PR when snapsyncing forest. The retry might be grouping together requests that are getting overlapping range responses. I will let it run for a few more hours and see if it makes any headway or if it is deadlocked.

edit: the deadlock did resolve, but it took a couple hours. Once resolved, sync progressed normally until it hit another problematic range. The test instance I used today spent ~6 hours failing and retrying every second, after 12 hours it is roughly 37% complete. So, 👍 it does progress, but it seems like there should be a better / faster strategy we can use.

is your sync finished correctly ?

[garyschulte]

is your sync finished correctly ?

It did finish worldstate sync, the heal was still running when the process was shut down but it looked like it was going to finish. 👍

[matkt]

@garyschulte do we want to keep log level to info ? or maybe adding a logic to log only one time in order to avoid spam but give us a way to debug and understand that retry occured

[garyschulte]

@garyschulte do we want to keep log level to info ? or maybe adding a logic to log only one time in order to avoid spam but give us a way to debug and understand that retry occured

Seems like a good idea to reduce or rate limit the logging. on the low iops forest node, the retry logs were once per second for several hours.

[matkt]

@garyschulte do we want to keep log level to info ? or maybe adding a logic to log only one time in order to avoid spam but give us a way to debug and understand that retry occured

Seems like a good idea to reduce or rate limit the logging. on the low iops forest node, the retry logs were once per second for several hours.

I modified the code to not diplay all issues, I will start a simple snapsync just to be sure and I think we will be able to merge it

[matkt]

Sync worked, Imo we can merge if you are ok

[garyschulte]

🚢