Bound finalized header, execution header and sync committee by pruning older data beyond threshold #810
Conversation
| while number_of_sync_committees_to_remove > 0 { | ||
| <SyncCommittees<T>>::remove(current_sync_committee_to_remove); | ||
| number_of_sync_committees_to_remove = | ||
| number_of_sync_committees_to_remove.saturating_sub(1); | ||
| current_sync_committee_to_remove = | ||
| current_sync_committee_to_remove.saturating_sub(1); | ||
| } | ||
| } |
There was a problem hiding this comment.
The problem here is the performance overhead of the while loop which could use up all the block weight.
There was a problem hiding this comment.
while loop would run only 1 time. As it runs after every insert.
| for _i in 0..execution_headers_to_delete { | ||
| let execution_header_hash = | ||
| <ExecutionHeadersMapping<T>>::get(oldest_mapping_to_delete); | ||
| <ExecutionHeadersMapping<T>>::remove(oldest_mapping_to_delete); | ||
| <ExecutionHeaders<T>>::remove(execution_header_hash); | ||
| oldest_mapping_to_delete += 1; | ||
| } |
There was a problem hiding this comment.
There was a problem hiding this comment.
while loop would run only 1 time. As it runs after every insert.
|
@ParthDesai Cool and thanks for the contribution! Though IMHO implementation with some ringbuffer structure without pruning could be more simple and effiecient. |
The The |
But, yes. The Ringbuffer structure does look like something which can be more efficient and simple than this. I will open another PR with updated implementation. |
Description
This PR enables pruning older
FinalizedHeader,ExecutionHeaderandSyncCommittee. This is required to put bounds on the storage of the pallet. As left unchecked it could be huge amount of data.First three PR adds support for pruning of
FinalizedHeader,ExecutionHeaderandSyncCommitteerespectively. Fourth commit adds comprehensive tests for the same. Fifth commit is just cargo fmt changes. And last commit introduces relayer changes.How it works?
ExecutionHeader
For
ExecutionHeaderthere is a queue like simulated data maintained that allows pruning of older data as the new data gets added.Since execution headers are not contiguous, to maintain the queue, we need to maintain a mapping of
u64 -> ExecutionHeaderand also keep track of front and back of the queue. While we push new data to back, we remove data from front this is supported by(oldest_bound, latest_bound)storage.FinalizedHeader
For
FinalizedHeaderwe are pruning it as perMaxFinalizedHeaderSlotArray. As,FinalizedHeaderand its associated data with slot less than0th element of the vector will not be used by relayer in anycase.SyncCommittee
Since
SyncCommitteeis stored in an order, we can rely on it to prune oldest sync committee.Configuration changes
Both
ExecutionHeaderandSyncCommittee's pruning threshold is defined in pallet configuration.Relayer changes
Due to change in
FinalizedSlotArraystorage, there is a change in relayer to maintain compatibility.