RocksDB Busy / Pipeline failed after 22.10.0 update and resumption of node 1000 blocks behind head

[non-fungible-nelson]

Description

As a user, I need my sync to resume correctly after some downtime for an upgrade without database issues so I can resume operation without a restart.

Acceptance Criteria

• Sync resumes after an update correctly
• Database is able to open
• No resync is needed

Steps to Reproduce (Bug)

1. User had a previously synced goerli node on an older version
2. Took node offline
3. Waited about an hour and updated node to 22.10.0
4. Node attempted to resume sync
5. Node gets RocksDB Busy error (see screenshot)
   

More logs in comment below.

Expected behavior: [What you expect to happen]
Node resumes sync normally and database is able to open

Actual behavior: [What actually happens]
Database is considered busy

Frequency: [What percentage of the time does it occur?]
Always on node restart

Versions (Add all that apply)

• Software version: 22.10.0

[non-fungible-nelson]

Might relate to @matkt's sync resumption mechanics

[non-fungible-nelson]

Could copy/relate fix from #3719

[non-fungible-nelson]

besu_export2.txt
Logs from user with DEBUG enabled.

[matkt]

the user seems to be using fastsync. if you can confirm. if so it is not related to this fix and I advise the user to switch to snapsync