Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Utility][Persistence][Consensus] Implement SavePoints & Rollbacks #562

Open
1 of 17 tasks
deblasis opened this issue Mar 8, 2023 · 3 comments
Open
1 of 17 tasks

[Utility][Persistence][Consensus] Implement SavePoints & Rollbacks #562

deblasis opened this issue Mar 8, 2023 · 3 comments
Assignees
@dylanlott
Labels
consensus Consensus specific changes core Core infrastructure - protocol related persistence Persistence specific changes utility Utility specific changes
Milestone
M1: Pocket PoS (P...

Comments

@deblasis
Copy link
Contributor

deblasis commented Mar 8, 2023 •
edited
Loading

Objective

This issue is going to be an "umbrella" ticket/epic used to wrap and link all the intermediate steps/deliverables that are necessary to build Savepoints and Rollbacks as highlighted in the Design Document - Issue #493

Origin Document

This is the result of R&D work on #508 which was essentially asking the assignee to build savepoints and rollbacks, just with a different wording thanks to a more "practical" problem to solve.

Goals

  • Implement an abstraction that can contain the operations performed on the various persistence datastores (as described in the design document) and apply/rollback them atomically
  • Implement a way to serialize and deserialize the whole persistence state so that's possible to create snapshots and rehydrate them
  • Implement the Moonshot parts of the design document (perhaps out-of-scope for now? I won't include the deliverables below but please let me know if I should)
  • Ensure that the state of the modules is restored in case of a rollback

Deliverable

Non-goals / Non-deliverables

  • Changes in logic with particular focus to Consensus

General issue deliverables

  • Update the appropriate CHANGELOG(s)
  • Update any relevant local/global README(s)
  • Update relevant source code tree explanations
  • Add or update any relevant or supporting mermaid diagrams

Testing Methodology

  • Task specific tests or benchmarks: make ...
  • New tests or benchmarks: make ...
  • All tests: make test_all
  • LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md

Creator: @deblasis
Co-Owners: @Olshansk
@deblasis deblasis added core Core infrastructure - protocol related consensus Consensus specific changes utility Utility specific changes persistence Persistence specific changes labels Mar 8, 2023
@deblasis deblasis self-assigned this Mar 8, 2023
This was referenced Mar 8, 2023
Open
Open
Open
Open
Open
Open
Open
Open
@Olshansk
Copy link
Member

Olshansk commented Mar 9, 2023 •
edited
Loading

Implement the Moonshot parts of the design document (perhaps out-of-scope for now? I won't include the deliverables below but please let me know if I should)

@deblasis This is a goal? Are you implying that it's a stretch goal that may be possible, but you don't want to make it one of the deliverables?

@deblasis
Copy link
Contributor Author

deblasis commented Mar 9, 2023

@deblasis This is a goal? Are you implying that it's a stretch goal that may be possible, but you don't want to make it one of the deliverables?

The goal was to have a discussion about it and here we are :) I was meant to raise this yesterday but I forgot.

I think that the moonshot part should be a separate issue in the backlog.

Thoughts?

@Olshansk
Copy link
Member

Olshansk commented Mar 9, 2023

I think that the moonshot part should be a separate issue in the backlog.

Agreed.

Side note: I accidentally just discovered that if I highlight part of your message and click "quote reply", it only quotes the part that I highlighted.

@deblasis deblasis mentioned this issue Mar 12, 2023
Merged
16 tasks
deblasis added a commit that referenced this issue Mar 26, 2023
@deblasis
[Utility][Persistence][Savepoints/Rollbacks] Refactor UtilityContext …
19bf4d3 
…into UtilityUnitOfWork (Issue #563) (#577)

## Description

This PR creates the logical abstraction that would represent the "Unit
Of Work" (inspired by [Martin Fowler's work in his "Patterns of
Enterprise Application
Architecture"](https://martinfowler.com/eaaCatalog/unitOfWork.html))

Essentially what previously was known as `UtilityContext` becomes
`LeaderUtilityUnitOfWork` or `ReplicaUtilityUnitOfWork` depending on the
fact that the current node is the Leader or not (in Consensus).

It touches many files because it moves everything that is interacting
with `Persistence` into the `utility/unit_of_work` folder.
This is to facilitate subsequent refactorings (see related PRs under
#562) into more modular components.

This PR also sets the ground up for the completion of #508 

## Issue

Fixes #563 

## Type of change

Please mark the relevant option(s):

- [x] New feature, functionality or library
- [ ] Bug fix
- [ ] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- Refactored `UtilityContext` to be a polymorphic `UtilityOfWork`
- Introduced `LeaderUtilityUnitOfWork` and `ReplicaUtilityUnitOfWork`
that expose the methods that are going be to used depending on the
"role" of the node in the consensus process.
- Moved `utilityContext` related code under `utility/unit_of_work` and
renamed all references accordingly

## Testing

- [x] `make develop_test`
- [x]
[LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md)
w/ all of the steps outlined in the `README`

<!-- REMOVE this comment block after following the instructions
 If you added additional tests or infrastructure, describe it here.
 Bonus points for images and videos or gifs.
-->

## Required Checklist

- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have tested my changes using the available tooling
- [x] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [x] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added, or updated,
[mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding
README(s)
- [ ] I have added, or updated, documentation and
[mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*`
if I updated `shared/*`README(s)

---------

Signed-off-by: Alessandro De Blasis <alex@deblasis.net>
@Olshansk Olshansk assigned dylanlott and unassigned deblasis Apr 19, 2023
@dylanlott dylanlott mentioned this issue May 23, 2023
Merged
20 tasks
dylanlott added a commit that referenced this issue May 24, 2023
@dylanlott
[Persistence] Refactors BlockStore Interface (#774)
f8da548 
## Description

This PR refactors the persistence module to create the BlockStore
interface. In doing so, it stops higher level components from directly
calling the KVStore and exposes specialized GetBlock and StoreBlock
functions instead. These two will eventually be wrapped with savepoints
and rollbacks logic to facilitate atomic commits to the block store.

## Issue

Parts of #562

## Type of change

Please mark the relevant option(s):

- [ ] New feature, functionality or library
- [ ] Bug fix
- [x] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- Creates the BlockStore interface and replaces mocks to use it where
possible
- Forces height and block serialization handling down into the
BlockStore module, making it cleaner and easier to get any block from
the bus.

## Testing

- [x] `make develop_test`; if any code changes were made
- [x] `make test_e2e` on [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any code changes were made
- [x] `e2e-devnet-test` passes tests on
[DevNet](https://pocketnetwork.notion.site/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd);
if any code was changed
- [ ] [Docker Compose
LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md);
if any major functionality was changed or introduced
- [x] [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any infrastructure or configuration changes were made

<!-- REMOVE this comment block after following the instructions
 If you added additional tests or infrastructure, describe it here.
 Bonus points for images and videos or gifs.
-->

## Required Checklist

- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added, or updated, [`godoc` format
comments](https://go.dev/blog/godoc) on touched members (see:
[tip.golang.org/doc/comment](https://tip.golang.org/doc/comment))
- [x] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [ ] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added, or updated,
[mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding
README(s)
- [ ] I have added, or updated, documentation and
[mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*`
if I updated `shared/*`README(s)
Olshansk pushed a commit that referenced this issue Jun 1, 2023
@dylanlott @Olshansk
[Persistence] Refactors BlockStore Interface (#774)
0e168d2 
## Description

This PR refactors the persistence module to create the BlockStore
interface. In doing so, it stops higher level components from directly
calling the KVStore and exposes specialized GetBlock and StoreBlock
functions instead. These two will eventually be wrapped with savepoints
and rollbacks logic to facilitate atomic commits to the block store.

## Issue

Parts of #562

## Type of change

Please mark the relevant option(s):

- [ ] New feature, functionality or library
- [ ] Bug fix
- [x] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- Creates the BlockStore interface and replaces mocks to use it where
possible
- Forces height and block serialization handling down into the
BlockStore module, making it cleaner and easier to get any block from
the bus.

## Testing

- [x] `make develop_test`; if any code changes were made
- [x] `make test_e2e` on [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any code changes were made
- [x] `e2e-devnet-test` passes tests on
[DevNet](https://pocketnetwork.notion.site/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd);
if any code was changed
- [ ] [Docker Compose
LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md);
if any major functionality was changed or introduced
- [x] [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any infrastructure or configuration changes were made

<!-- REMOVE this comment block after following the instructions
 If you added additional tests or infrastructure, describe it here.
 Bonus points for images and videos or gifs.
-->

## Required Checklist

- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added, or updated, [`godoc` format
comments](https://go.dev/blog/godoc) on touched members (see:
[tip.golang.org/doc/comment](https://tip.golang.org/doc/comment))
- [x] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [ ] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added, or updated,
[mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding
README(s)
- [ ] I have added, or updated, documentation and
[mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*`
if I updated `shared/*`README(s)
@dylanlott dylanlott mentioned this issue Jun 6, 2023
Merged
18 tasks
dylanlott added a commit that referenced this issue Jun 21, 2023
@dylanlott @Olshansk
[Persistence] TreeStore Refactor (#756)
70a1a0e 
## Description

Refactors the stateTrees component behind the TreeStore interface.

## Issue

Part of a refactor effort towards implementing #562

Part of a series of refactors in the persistence package, with #736
being the first.

## Type of change

Please mark the relevant option(s):

- [ ] New feature, functionality or library
- [ ] Bug fix
- [x] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- Pulls the stateTrees component down into its own component behind a
dedicated interface for managing the SMT store.

## Testing

- [x] `make develop_test`; if any code changes were made
- [ ] [Docker Compose
LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md);
if any major functionality was changed or introduced
- [x] [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any infrastructure or configuration changes were made

## Required Checklist

- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added, or updated, [`godoc` format
comments](https://go.dev/blog/godoc) on touched members (see:
[tip.golang.org/doc/comment](https://tip.golang.org/doc/comment))
- [ ] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [ ] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added, or updated,
[mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding
README(s)
- [ ] I have added, or updated, documentation and
[mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*`
if I updated `shared/*`README(s)

---------

Co-authored-by: Daniel Olshansky <olshansky.daniel@gmail.com>
Co-authored-by: Daniel Olshansky <olshansky@pokt.network>
This was referenced Jul 11, 2023
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dylanlott dylanlott

Labels
consensus Consensus specific changes core Core infrastructure - protocol related persistence Persistence specific changes utility Utility specific changes
Projects
V1 Dashboard
Status: In Review
Milestone
M1: Pocket PoS (Proof of Stake)
Development

No branches or pull requests
3 participants
@Olshansk @dylanlott @deblasis