Skip to content

Commit

Permalink
Inner hashing of value in state trie (runtime versioning). (paritytec…
Browse files Browse the repository at this point in the history
…h#9732)

* starting

* Updated from other branch.

* setting flag

* flag in storage struct

* fix flagging to access and insert.

* added todo to fix

* also missing serialize meta to storage proof

* extract meta.

* Isolate old trie layout.

* failing test that requires storing in meta when old hash scheme is used.

* old hash compatibility

* Db migrate.

* runing tests with both states when interesting.

* fix chain spec test with serde default.

* export state (missing trie function).

* Pending using new branch, lacking genericity on layout resolution.

* extract and set global meta

* Update to branch 4

* fix iterator with root flag (no longer insert node).

* fix trie root hashing of root

* complete basic backend.

* Remove old_hash meta from proof that do not use inner_hashing.

* fix trie test for empty (force layout on empty deltas).

* Root update fix.

* debug on meta

* Use trie key iteration that do not include value in proofs.

* switch default test ext to use inner hash.

* small integration test, and fix tx cache mgmt in ext.
test  failing

* Proof scenario at state-machine level.

* trace for db upgrade

* try different param

* act more like iter_from.

* Bigger batches.

* Update trie dependency.

* drafting codec changes and refact

* before removing unused branch no value alt hashing.
more work todo rename all flag var to alt_hash, and remove extrinsic
replace by storage query at every storage_root call.

* alt hashing only for branch with value.

* fix trie tests

* Hash of value include the encoded size.

* removing fields(broken)

* fix trie_stream to also include value length in inner hash.

* triedbmut only using alt type if inner hashing.

* trie_stream to also only use alt hashing type when actually alt hashing.

* Refactor meta state, logic should work with change of trie treshold.

* Remove NoMeta variant.

* Remove state_hashed trigger specific functions.

* pending switching to using threshold, new storage root api does not
make much sense.

* refactoring to use state from backend (not possible payload changes).

* Applying from previous state

* Remove default from storage, genesis need a special build.

* rem empty space

* Catch problem: when using triedb with default: we should not revert
nodes: otherwhise thing as trie codec cannot decode-encode without
changing state.

* fix compilation

* Right logic to avoid switch on reencode when default layout.

* Clean up some todos

* remove trie meta from root upstream

* update upstream and fix benches.

* split some long lines.

* UPdate trie crate to work with new design.

* Finish update to refactored upstream.

* update to latest triedb changes.

* Clean up.

* fix executor test.

* rust fmt from master.

* rust format.

* rustfmt

* fix

* start host function driven versioning

* update state-machine part

* still need access to state version from runtime

* state hash in mem: wrong

* direction likely correct, but passing call to code exec for genesis
init seem awkward.

* state version serialize in runtime, wrong approach, just initialize it
with no threshold for core api < 4 seems more proper.

* stateversion from runtime version (core api >= 4).

* update trie, fix tests

* unused import

* clean some TODOs

* Require RuntimeVersionOf for executor

* use RuntimeVersionOf to resolve genesis state version.

* update runtime version test

* fix state-machine tests

* TODO

* Use runtime version from storage wasm with fast sync.

* rustfmt

* fmt

* fix test

* revert useless changes.

* clean some unused changes

* fmt

* removing useless trait function.

* remove remaining reference to state_hash

* fix some imports

* Follow chain state version management.

* trie update, fix and constant threshold for trie layouts.

* update deps

* Update to latest trie pr changes.

* fix benches

* Verify proof requires right layout.

* update trie_root

* Update trie deps to  latest

* Update to latest trie versioning

* Removing patch

* update lock

* extrinsic for sc-service-test using layout v0.

* Adding RuntimeVersionOf to CallExecutor works.

* fmt

* error when resolving version and no wasm in storage.

* use existing utils to instantiate runtime code.

* Patch to delay runtime switch.

* Revert "Patch to delay runtime switch."

This reverts commit d35f273.

* useless closure

* remove remaining state_hash variables.

* Remove outdated comment

* useless inner hash

* fmt

* fmt and opt-in feature to apply state change.

* feature gate core version, use new test feature for node and test node

* Use a 'State' api version instead of Core one.

* fix merge of test function

* use blake macro.

* Fix state api (require declaring the api in runtime).

* Opt out feature, fix macro for io to select a given version
instead of latest.

* run test nodes on new state.

* fix

* Apply review change (docs and error).

* fmt

* use explicit runtime_interface in doc test

* fix ui test

* fix doc test

* fmt

* use default for path and specname when resolving version.

* small review related changes.

* doc value size requirement.

* rename old_state feature

* Remove macro changes

* feature rename

* state version as host function parameter

* remove flag for client api

* fix tests

* switch storage chain proof to V1

* host functions, pass by state version enum

* use WrappedRuntimeCode

* start

* state_version in runtime version

* rust fmt

* Update storage proof of max size.

* fix runtime version rpc test

* right intent of convert from compat

* fix doc test

* fix doc test

* split proof

* decode without replay, and remove some reexports.

* Decode with compatibility by default.

* switch state_version to u8. And remove RuntimeVersionBasis.

* test

* use api when reading embedded version

* fix decode with apis

* extract core version instead

* test fix

* unused import

* review changes.

Co-authored-by: kianenigma <kian@parity.io>
  • Loading branch information
cheme and kianenigma authored Dec 24, 2021
1 parent 026ec33 commit b03e8bc
Show file tree
Hide file tree
Showing 78 changed files with 1,811 additions and 779 deletions.
34 changes: 23 additions & 11 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions bin/node-template/runtime/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ pub const VERSION: RuntimeVersion = RuntimeVersion {
impl_version: 1,
apis: RUNTIME_API_VERSIONS,
transaction_version: 1,
state_version: 1,
};

/// This determines the average expected block time that we are targeting.
Expand Down
5 changes: 2 additions & 3 deletions bin/node/bench/src/generator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ use std::{collections::HashMap, sync::Arc};

use kvdb::KeyValueDB;
use node_primitives::Hash;
use sp_trie::{trie_types::TrieDBMut, TrieMut};
use sp_trie::{trie_types::TrieDBMutV1, TrieMut};

use crate::simple_trie::SimpleTrie;

Expand All @@ -43,8 +43,7 @@ pub fn generate_trie(
);
let mut trie = SimpleTrie { db, overlay: &mut overlay };
{
let mut trie_db = TrieDBMut::new(&mut trie, &mut root);

let mut trie_db = TrieDBMutV1::<crate::simple_trie::Hasher>::new(&mut trie, &mut root);
for (key, value) in key_values {
trie_db.insert(&key, &value).expect("trie insertion failed");
}
Expand Down
6 changes: 3 additions & 3 deletions bin/node/bench/src/trie.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ use kvdb::KeyValueDB;
use lazy_static::lazy_static;
use rand::Rng;
use sp_state_machine::Backend as _;
use sp_trie::{trie_types::TrieDBMut, TrieMut as _};
use sp_trie::{trie_types::TrieDBMutV1, TrieMut as _};
use std::{borrow::Cow, collections::HashMap, sync::Arc};

use node_primitives::Hash;
Expand Down Expand Up @@ -286,8 +286,8 @@ impl core::Benchmark for TrieWriteBenchmark {

let mut overlay = HashMap::new();
let mut trie = SimpleTrie { db: kvdb.clone(), overlay: &mut overlay };
let mut trie_db_mut =
TrieDBMut::from_existing(&mut trie, &mut new_root).expect("Failed to create TrieDBMut");
let mut trie_db_mut = TrieDBMutV1::from_existing(&mut trie, &mut new_root)
.expect("Failed to create TrieDBMut");

for (warmup_key, warmup_value) in self.warmup_keys.iter() {
let value = trie_db_mut
Expand Down
4 changes: 2 additions & 2 deletions bin/node/executor/benches/bench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,14 +83,14 @@ fn construct_block<E: Externalities>(
parent_hash: Hash,
extrinsics: Vec<CheckedExtrinsic>,
) -> (Vec<u8>, Hash) {
use sp_trie::{trie_types::Layout, TrieConfiguration};
use sp_trie::{LayoutV0, TrieConfiguration};

// sign extrinsics.
let extrinsics = extrinsics.into_iter().map(sign).collect::<Vec<_>>();

// calculate the header fields that we can.
let extrinsics_root =
Layout::<BlakeTwo256>::ordered_trie_root(extrinsics.iter().map(Encode::encode))
LayoutV0::<BlakeTwo256>::ordered_trie_root(extrinsics.iter().map(Encode::encode))
.to_fixed_bytes()
.into();

Expand Down
2 changes: 1 addition & 1 deletion bin/node/executor/tests/common.rs
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ pub fn construct_block(
extrinsics: Vec<CheckedExtrinsic>,
babe_slot: Slot,
) -> (Vec<u8>, Hash) {
use sp_trie::{trie_types::Layout, TrieConfiguration};
use sp_trie::{LayoutV1 as Layout, TrieConfiguration};

// sign extrinsics.
let extrinsics = extrinsics.into_iter().map(sign).collect::<Vec<_>>();
Expand Down
1 change: 1 addition & 0 deletions bin/node/runtime/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ pub const VERSION: RuntimeVersion = RuntimeVersion {
impl_version: 0,
apis: RUNTIME_API_VERSIONS,
transaction_version: 2,
state_version: 1,
};

/// The BABE epoch configuration at genesis.
Expand Down
9 changes: 7 additions & 2 deletions client/api/src/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ use sp_core::offchain::OffchainStorage;
use sp_runtime::{
generic::BlockId,
traits::{Block as BlockT, HashFor, NumberFor},
Justification, Justifications, Storage,
Justification, Justifications, StateVersion, Storage,
};
use sp_state_machine::{
ChildStorageCollection, IndexOperation, OffchainChangesCollection, StorageCollection,
Expand Down Expand Up @@ -166,10 +166,15 @@ pub trait BlockImportOperation<Block: BlockT> {
&mut self,
storage: Storage,
commit: bool,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash>;

/// Inject storage data into the database replacing any existing data.
fn reset_storage(&mut self, storage: Storage) -> sp_blockchain::Result<Block::Hash>;
fn reset_storage(
&mut self,
storage: Storage,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash>;

/// Set storage changes.
fn update_storage(
Expand Down
4 changes: 2 additions & 2 deletions client/api/src/call_executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
//! A method call executor interface.

use codec::{Decode, Encode};
use sc_executor::RuntimeVersion;
use sc_executor::{RuntimeVersion, RuntimeVersionOf};
use sp_core::NativeOrEncoded;
use sp_externalities::Extensions;
use sp_runtime::{generic::BlockId, traits::Block as BlockT};
Expand All @@ -42,7 +42,7 @@ pub trait ExecutorProvider<Block: BlockT> {
}

/// Method call executor.
pub trait CallExecutor<B: BlockT> {
pub trait CallExecutor<B: BlockT>: RuntimeVersionOf {
/// Externalities error type.
type Error: sp_state_machine::Error;

Expand Down
15 changes: 11 additions & 4 deletions client/api/src/in_mem.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ use sp_core::{
use sp_runtime::{
generic::BlockId,
traits::{Block as BlockT, HashFor, Header as HeaderT, NumberFor, Zero},
Justification, Justifications, Storage,
Justification, Justifications, StateVersion, Storage,
};
use sp_state_machine::{
Backend as StateBackend, ChildStorageCollection, InMemoryBackend, IndexOperation,
Expand Down Expand Up @@ -506,6 +506,7 @@ where
&mut self,
storage: Storage,
commit: bool,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash> {
check_genesis_storage(&storage)?;

Expand All @@ -519,6 +520,7 @@ where
let (root, transaction) = self.old_state.full_storage_root(
storage.top.iter().map(|(k, v)| (k.as_ref(), Some(v.as_ref()))),
child_delta,
state_version,
);

if commit {
Expand Down Expand Up @@ -566,12 +568,17 @@ where
&mut self,
storage: Storage,
commit: bool,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash> {
self.apply_storage(storage, commit)
self.apply_storage(storage, commit, state_version)
}

fn reset_storage(&mut self, storage: Storage) -> sp_blockchain::Result<Block::Hash> {
self.apply_storage(storage, true)
fn reset_storage(
&mut self,
storage: Storage,
state_version: StateVersion,
) -> sp_blockchain::Result<Block::Hash> {
self.apply_storage(storage, true, state_version)
}

fn insert_aux<I>(&mut self, ops: I) -> sp_blockchain::Result<()>
Expand Down
1 change: 1 addition & 0 deletions client/block-builder/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,7 @@ where
header.extrinsics_root().clone(),
HashFor::<Block>::ordered_trie_root(
self.extrinsics.iter().map(Encode::encode).collect(),
sp_runtime::StateVersion::V0,
),
);

Expand Down
13 changes: 9 additions & 4 deletions client/db/src/bench.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ use sp_core::{
};
use sp_runtime::{
traits::{Block as BlockT, HashFor},
Storage,
StateVersion, Storage,
};
use sp_state_machine::{
backend::Backend as StateBackend, ChildStorageCollection, DBValue, ProofRecorder,
Expand Down Expand Up @@ -73,6 +73,7 @@ impl<Block: BlockT> sp_state_machine::Storage<HashFor<Block>> for StorageDb<Bloc
}
}
}

/// State that manages the backend database reference. Allows runtime to control the database.
pub struct BenchmarkingState<B: BlockT> {
root: Cell<B::Hash>,
Expand Down Expand Up @@ -105,9 +106,10 @@ impl<B: BlockT> BenchmarkingState<B> {
record_proof: bool,
enable_tracking: bool,
) -> Result<Self, String> {
let state_version = sp_runtime::StateVersion::default();
let mut root = B::Hash::default();
let mut mdb = MemoryDB::<HashFor<B>>::default();
sp_state_machine::TrieDBMut::<HashFor<B>>::new(&mut mdb, &mut root);
sp_state_machine::TrieDBMutV1::<HashFor<B>>::new(&mut mdb, &mut root);

let mut state = BenchmarkingState {
state: RefCell::new(None),
Expand Down Expand Up @@ -138,6 +140,7 @@ impl<B: BlockT> BenchmarkingState<B> {
state.state.borrow_mut().as_mut().unwrap().full_storage_root(
genesis.top.iter().map(|(k, v)| (k.as_ref(), Some(v.as_ref()))),
child_delta,
state_version,
);
state.genesis = transaction.clone().drain();
state.genesis_root = root.clone();
Expand Down Expand Up @@ -415,28 +418,30 @@ impl<B: BlockT> StateBackend<HashFor<B>> for BenchmarkingState<B> {
fn storage_root<'a>(
&self,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, Self::Transaction)
where
B::Hash: Ord,
{
self.state
.borrow()
.as_ref()
.map_or(Default::default(), |s| s.storage_root(delta))
.map_or(Default::default(), |s| s.storage_root(delta, state_version))
}

fn child_storage_root<'a>(
&self,
child_info: &ChildInfo,
delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
state_version: StateVersion,
) -> (B::Hash, bool, Self::Transaction)
where
B::Hash: Ord,
{
self.state
.borrow()
.as_ref()
.map_or(Default::default(), |s| s.child_storage_root(child_info, delta))
.map_or(Default::default(), |s| s.child_storage_root(child_info, delta, state_version))
}

fn pairs(&self) -> Vec<(Vec<u8>, Vec<u8>)> {
Expand Down
Loading

0 comments on commit b03e8bc

Please sign in to comment.