Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bonsai snapshot worldstate #4409

Merged

Conversation

garyschulte
Copy link
Contributor

@garyschulte garyschulte commented Sep 16, 2022

PR description

This PR adds support for using rocksdb snapshot transactions as the basis for a bonsai worldstate. This gives us the ability to have an isolated copy of the world state for concurrent operations like block production, block validation, transaction validation, historical state queries, etc.

The notion is to merge this capability into main, and start replacing the uses of BonsaiLayeredWorldState and direct uses of BonsaiInMemoryWorldState when we need isolated and/or concurrent operations on the worldstate.

This is a strictly additive PR and does not change existing implementations. Uses of snapshot worldstates will be useful to resolve issues such as #4372 #4250 #4199 #4151 and other bonsai concurrency issues.

Changes in support of snapshot worldstate include:

  • moving (back) to OptimisticTransactionDB, so we can mutate the snapshot transactions
  • creating BonsaiWorldStateKeyValueStorage.BonsaiUpdater interface extending WorldStateStorage.Updater, so we can have a snapshot specific Updater implementation
  • Addition of SnappableStorage, SnappedStorage to plugin-api in order to support/expose the snapshot behavior

Fixed Issue(s)

relates to #4402
relates to #4403

Documentation

  • I thought about documentation and added the doc-change-required label to this PR if
    updates are required.

Changelog

@matkt
Copy link
Contributor

matkt commented Sep 17, 2022

Excellent work I will review it on Monday or this weekend

Copy link
Contributor

@matkt matkt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will continue the review but i already share this

}

public static class SnapshotUpdater implements BonsaiWorldStateKeyValueStorage.BonsaiUpdater {
// private static final Logger LOG =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you missed this line

Copy link
Contributor

@matkt matkt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to be ok. waiting for the next PR related to snapshot

@garyschulte garyschulte force-pushed the feature/snapshot-worldstate-reprise branch 2 times, most recently from d1edc6b to 12c045c Compare October 10, 2022 18:28
@garyschulte garyschulte marked this pull request as draft October 11, 2022 00:04
@garyschulte garyschulte force-pushed the feature/snapshot-worldstate-reprise branch from 01725fe to c675a28 Compare October 11, 2022 17:08
@garyschulte garyschulte marked this pull request as ready for review October 11, 2022 19:29
@garyschulte garyschulte marked this pull request as draft October 11, 2022 22:19
@garyschulte garyschulte force-pushed the feature/snapshot-worldstate-reprise branch from c675a28 to 158937a Compare October 12, 2022 21:36
@garyschulte garyschulte marked this pull request as ready for review October 13, 2022 14:41
@garyschulte
Copy link
Contributor Author

IMO, this is ready to merge to main to clear the way for subsequent PRs that implement bonsai snapshots.

@garyschulte garyschulte force-pushed the feature/snapshot-worldstate-reprise branch 2 times, most recently from 4fe232b to 04a0820 Compare October 13, 2022 15:04
this.trieBranchSnap = (SnappedKeyValueStorage) snapshotWorldStateStorage.trieBranchStorage;
}

public static BonsaiSnapshotWorldState create(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a potential race with the storages being changed while the snapshots are generated?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some risk of that theoretically, but considering we write to rocksdb in batch commits and the snapshots are essentially atomic, this is probably very very minimal chance in practice. There is certainly far less concurrency risk than our current strategy where the underlying persisted state can change underneath a BonsaiInMemoryWorldState or BonsaiLayeredWorldState

Copy link
Contributor Author

@garyschulte garyschulte Oct 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems reasonable to implement locking in BonsaiPersistedWorldState so that we do not commit a batch write (via calculateRootHash) and take a snapshot concurrently. It should be a very short lock for getting a snapshot, and delaying getting a snapshot while waiting on a lock to release from a batch write would normally not be significant.

I will add this in one of the subsequent snapshot PRs and add tests 👍

} catch (final RocksDBException e) {
if (e.getMessage().contains(NO_SPACE_LEFT_ON_DEVICE)) {
LOG.error(e.getMessage());
System.exit(0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

System.exit feels too brutal to me. Is there any chance this can corrupt db for instance in case an ongoing update is happening in parallel? Maybe we don't access db from different threads.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. This is the same behavior as RocksDBTransaction. The right course of action isn't really clear since we do not have any disk space to work with.

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Isolation is working, but unclear whether trielogs are persisting correctly since we cannot create a snapshot and roll it to a particular hash in unit tests

Signed-off-by: garyschulte <garyschulte@gmail.com>
snapshot trieLog is persistable
snapshot roll forward/back is FAILING

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
…correctly dispose of snapshots

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
… extend persisted worldstate rather than in-memory worldstate

Signed-off-by: garyschulte <garyschulte@gmail.com>
close snapshot worldstate transactions on close
release snapshot on close of snapshot transaction (potentially fraught)

Signed-off-by: garyschulte <garyschulte@gmail.com>
@garyschulte garyschulte force-pushed the feature/snapshot-worldstate-reprise branch from 04a0820 to 947ae51 Compare October 13, 2022 16:21
@garyschulte garyschulte enabled auto-merge (squash) October 13, 2022 16:28
@garyschulte garyschulte merged commit d73ce21 into hyperledger:main Oct 13, 2022
macfarla pushed a commit to jflo/besu that referenced this pull request Jan 10, 2023
* use optimistictransactiondb for mutable isolated snapshots
* plumbing necessary to have a snapshot specific updater.
* snapshot rolling working
* implement AutoCloseable on BonsaiSnapshotWorldState to ensure we can correctly dispose of snapshots
* add snapshot transaction cloning, change snapshot based worldstate to extend persisted worldstate rather than in-memory worldstate

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
eum602 pushed a commit to lacchain/besu that referenced this pull request Nov 3, 2023
* use optimistictransactiondb for mutable isolated snapshots
* plumbing necessary to have a snapshot specific updater.
* snapshot rolling working
* implement AutoCloseable on BonsaiSnapshotWorldState to ensure we can correctly dispose of snapshots
* add snapshot transaction cloning, change snapshot based worldstate to extend persisted worldstate rather than in-memory worldstate

Signed-off-by: garyschulte <garyschulte@gmail.com>
@garyschulte garyschulte deleted the feature/snapshot-worldstate-reprise branch January 24, 2024 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants