Persistent State Storage for Improved State Syncing #4
Labels
I5-enhancement
An additional feature request.
Comments
This was referenced Jun 3, 2023
the-right-joyce
added
I5-enhancement
2 tasks
fixxxedpoint
pushed a commit
to fixxxedpoint/polkadot-sdk
that referenced
this issue
Jun 19, 2024
…) (paritytech#4) This PR is fixing a bug in the sync mechanism between wasmi and pallet-contracts. This bug leads to essentially double charging all the gas that was used during the execution of the host function. When the `call` host function is used for recursion this will lead to a quadratic amount of gas consumption with regard to the nesting depth.We also took the chance to refactor the code in question and improve the rust docs. The bug was caused by not updating `GasMeter::executor_consumed` (previously `engine_consumed`) when leaving the host function. This lead to the value being stale (too low) when entering another host function. --------- Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: PG Herveou <pgherveou@gmail.com>
2 tasks
2 tasks
liuchengxu
added a commit
to subcoin-project/Grants-Program
that referenced
this issue
Aug 22, 2024
In light of recent developments, it has become evident that fully syncing to the tip of the Bitcoin network and enabling new nodes to perform fast sync to the latest Bitcoin state is more challenging than initially anticipated, caused by the huge state of UTXO set (over 12GiB). As a result, I propose adjusting the delivery goal for this milestone. The most significant known blocker is paritytech/polkadot-sdk#4. Other underlying issues may also contribute to the difficulty. Recent experiments have shown that fast sync from around block height 580,000 is currently infeasible, succeeding only on machines with 128GiB of memory (paritytech/polkadot-sdk#5053 (comment)), which is impractical for most users. Nevertheless, we have successfully demonstrated that decentralized fast sync is possible within a prototype implementation. While syncing to the Bitcoin network's tip remains a future target, addressing the existing technical challenges will require substantial R&D efforts. We remain committed to exploring potential solutions, including architectural changes and contributing to resolving issue paritytech/polkadot-sdk#4,
Noc2
pushed a commit
to w3f/Grants-Program
that referenced
this issue
Aug 23, 2024
In light of recent developments, it has become evident that fully syncing to the tip of the Bitcoin network and enabling new nodes to perform fast sync to the latest Bitcoin state is more challenging than initially anticipated, caused by the huge state of UTXO set (over 12GiB). As a result, I propose adjusting the delivery goal for this milestone. The most significant known blocker is paritytech/polkadot-sdk#4. Other underlying issues may also contribute to the difficulty. Recent experiments have shown that fast sync from around block height 580,000 is currently infeasible, succeeding only on machines with 128GiB of memory (paritytech/polkadot-sdk#5053 (comment)), which is impractical for most users. Nevertheless, we have successfully demonstrated that decentralized fast sync is possible within a prototype implementation. While syncing to the Bitcoin network's tip remains a future target, addressing the existing technical challenges will require substantial R&D efforts. We remain committed to exploring potential solutions, including architectural changes and contributing to resolving issue paritytech/polkadot-sdk#4,
liuchengxu
added a commit
to subcoin-project/polkadot-sdk
that referenced
this issue
Sep 20, 2024
* Set flag --execute-block as true by default * Ignore *.log * Finalize blocks with enough confirmations It's observed that the memory usage could be extremely high without the finalization, when the chain grows to 220000+. Concretely, the culprit of the high memory usage is creating `NonCanonicalOverlay`. There are also a few other improvements to import-blocks command.
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, during the state syncing process from the network, our node retains its internal state, including downloaded portions of the blockchain state and the target block, in memory. This poses a significant challenge: if the node restarts mid-sync, all the data must be re-downloaded.
To improve this process, I propose persistently storing the internal state outside of memory. This could be in a database or a plain file, allowing us to preserve the internal state across node restarts. This would be especially beneficial when handling a large blockchain state, which doesn't need to remain in memory.
However, one potential issue to consider is if the node is offline for an extended period, the state of the block may no longer be available on full nodes within the network, but only on archive nodes. To mitigate this, we could consider the implementation suggested in Issue paritytech/polkadot-sdk#523.
The text was updated successfully, but these errors were encountered: