Inner hashing of value in state trie (runtime versioning). (#9732)

* starting * Updated from other branch. * setting flag * flag in storage struct * fix flagging to access and insert. * added todo to fix * also missing serialize meta to storage proof * extract meta. * Isolate old trie layout. * failing test that requires storing in meta when old hash scheme is used. * old hash compatibility * Db migrate. * runing tests with both states when interesting. * fix chain spec test with serde default. * export state (missing trie function). * Pending using new branch, lacking genericity on layout resolution. * extract and set global meta * Update to branch 4 * fix iterator with root flag (no longer insert node). * fix trie root hashing of root * complete basic backend. * Remove old_hash meta from proof that do not use inner_hashing. * fix trie test for empty (force layout on empty deltas). * Root update fix. * debug on meta * Use trie key iteration that do not include value in proofs. * switch default test ext to use inner hash. * small integration test, and fix tx cache mgmt in ext. test failing * Proof scenario at state-machine level. * trace for db upgrade * try different param * act more like iter_from. * Bigger batches. * Update trie dependency. * drafting codec changes and refact * before removing unused branch no value alt hashing. more work todo rename all flag var to alt_hash, and remove extrinsic replace by storage query at every storage_root call. * alt hashing only for branch with value. * fix trie tests * Hash of value include the encoded size. * removing fields(broken) * fix trie_stream to also include value length in inner hash. * triedbmut only using alt type if inner hashing. * trie_stream to also only use alt hashing type when actually alt hashing. * Refactor meta state, logic should work with change of trie treshold. * Remove NoMeta variant. * Remove state_hashed trigger specific functions. * pending switching to using threshold, new storage root api does not make much sense. * refactoring to use state from backend (not possible payload changes). * Applying from previous state * Remove default from storage, genesis need a special build. * rem empty space * Catch problem: when using triedb with default: we should not revert nodes: otherwhise thing as trie codec cannot decode-encode without changing state. * fix compilation * Right logic to avoid switch on reencode when default layout. * Clean up some todos * remove trie meta from root upstream * update upstream and fix benches. * split some long lines. * UPdate trie crate to work with new design. * Finish update to refactored upstream. * update to latest triedb changes. * Clean up. * fix executor test. * rust fmt from master. * rust format. * rustfmt * fix * start host function driven versioning * update state-machine part * still need access to state version from runtime * state hash in mem: wrong * direction likely correct, but passing call to code exec for genesis init seem awkward. * state version serialize in runtime, wrong approach, just initialize it with no threshold for core api < 4 seems more proper. * stateversion from runtime version (core api >= 4). * update trie, fix tests * unused import * clean some TODOs * Require RuntimeVersionOf for executor * use RuntimeVersionOf to resolve genesis state version. * update runtime version test * fix state-machine tests * TODO * Use runtime version from storage wasm with fast sync. * rustfmt * fmt * fix test * revert useless changes. * clean some unused changes * fmt * removing useless trait function. * remove remaining reference to state_hash * fix some imports * Follow chain state version management. * trie update, fix and constant threshold for trie layouts. * update deps * Update to latest trie pr changes. * fix benches * Verify proof requires right layout. * update trie_root * Update trie deps to latest * Update to latest trie versioning * Removing patch * update lock * extrinsic for sc-service-test using layout v0. * Adding RuntimeVersionOf to CallExecutor works. * fmt * error when resolving version and no wasm in storage. * use existing utils to instantiate runtime code. * Patch to delay runtime switch. * Revert "Patch to delay runtime switch." This reverts commit d35f273. * useless closure * remove remaining state_hash variables. * Remove outdated comment * useless inner hash * fmt * fmt and opt-in feature to apply state change. * feature gate core version, use new test feature for node and test node * Use a 'State' api version instead of Core one. * fix merge of test function * use blake macro. * Fix state api (require declaring the api in runtime). * Opt out feature, fix macro for io to select a given version instead of latest. * run test nodes on new state. * fix * Apply review change (docs and error). * fmt * use explicit runtime_interface in doc test * fix ui test * fix doc test * fmt * use default for path and specname when resolving version. * small review related changes. * doc value size requirement. * rename old_state feature * Remove macro changes * feature rename * state version as host function parameter * remove flag for client api * fix tests * switch storage chain proof to V1 * host functions, pass by state version enum * use WrappedRuntimeCode * start * state_version in runtime version * rust fmt * Update storage proof of max size. * fix runtime version rpc test * right intent of convert from compat * fix doc test * fix doc test * split proof * decode without replay, and remove some reexports. * Decode with compatibility by default. * switch state_version to u8. And remove RuntimeVersionBasis. * test * use api when reading embedded version * fix decode with apis * extract core version instead * test fix * unused import * review changes. Co-authored-by: kianenigma <kian@parity.io>
paritytech · Dec 24, 2021 · b03e8bc · b03e8bc
1 parent 026ec33
commit b03e8bc
Show file tree

Hide file tree

Showing 78 changed files with 1,811 additions and 779 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/bin/node-template/runtime/src/lib.rs b/bin/node-template/runtime/src/lib.rs
@@ -102,6 +102,7 @@ pub const VERSION: RuntimeVersion = RuntimeVersion {
 	impl_version: 1,
 	apis: RUNTIME_API_VERSIONS,
 	transaction_version: 1,
+	state_version: 1,
 };
 
 /// This determines the average expected block time that we are targeting.

diff --git a/bin/node/bench/src/generator.rs b/bin/node/bench/src/generator.rs
@@ -20,7 +20,7 @@ use std::{collections::HashMap, sync::Arc};
 
 use kvdb::KeyValueDB;
 use node_primitives::Hash;
-use sp_trie::{trie_types::TrieDBMut, TrieMut};
+use sp_trie::{trie_types::TrieDBMutV1, TrieMut};
 
 use crate::simple_trie::SimpleTrie;
 
@@ -43,8 +43,7 @@ pub fn generate_trie(
 		);
 		let mut trie = SimpleTrie { db, overlay: &mut overlay };
 		{
-			let mut trie_db = TrieDBMut::new(&mut trie, &mut root);
-
+			let mut trie_db = TrieDBMutV1::<crate::simple_trie::Hasher>::new(&mut trie, &mut root);
 			for (key, value) in key_values {
 				trie_db.insert(&key, &value).expect("trie insertion failed");
 			}

diff --git a/bin/node/bench/src/trie.rs b/bin/node/bench/src/trie.rs
@@ -23,7 +23,7 @@ use kvdb::KeyValueDB;
 use lazy_static::lazy_static;
 use rand::Rng;
 use sp_state_machine::Backend as _;
-use sp_trie::{trie_types::TrieDBMut, TrieMut as _};
+use sp_trie::{trie_types::TrieDBMutV1, TrieMut as _};
 use std::{borrow::Cow, collections::HashMap, sync::Arc};
 
 use node_primitives::Hash;
@@ -286,8 +286,8 @@ impl core::Benchmark for TrieWriteBenchmark {
 
 		let mut overlay = HashMap::new();
 		let mut trie = SimpleTrie { db: kvdb.clone(), overlay: &mut overlay };
-		let mut trie_db_mut =
-			TrieDBMut::from_existing(&mut trie, &mut new_root).expect("Failed to create TrieDBMut");
+		let mut trie_db_mut = TrieDBMutV1::from_existing(&mut trie, &mut new_root)
+			.expect("Failed to create TrieDBMut");
 
 		for (warmup_key, warmup_value) in self.warmup_keys.iter() {
 			let value = trie_db_mut

diff --git a/bin/node/executor/benches/bench.rs b/bin/node/executor/benches/bench.rs
@@ -83,14 +83,14 @@ fn construct_block<E: Externalities>(
 	parent_hash: Hash,
 	extrinsics: Vec<CheckedExtrinsic>,
 ) -> (Vec<u8>, Hash) {
-	use sp_trie::{trie_types::Layout, TrieConfiguration};
+	use sp_trie::{LayoutV0, TrieConfiguration};
 
 	// sign extrinsics.
 	let extrinsics = extrinsics.into_iter().map(sign).collect::<Vec<_>>();
 
 	// calculate the header fields that we can.
 	let extrinsics_root =
-		Layout::<BlakeTwo256>::ordered_trie_root(extrinsics.iter().map(Encode::encode))
+		LayoutV0::<BlakeTwo256>::ordered_trie_root(extrinsics.iter().map(Encode::encode))
 			.to_fixed_bytes()
 			.into();
 

diff --git a/bin/node/executor/tests/common.rs b/bin/node/executor/tests/common.rs
@@ -142,7 +142,7 @@ pub fn construct_block(
 	extrinsics: Vec<CheckedExtrinsic>,
 	babe_slot: Slot,
 ) -> (Vec<u8>, Hash) {
-	use sp_trie::{trie_types::Layout, TrieConfiguration};
+	use sp_trie::{LayoutV1 as Layout, TrieConfiguration};
 
 	// sign extrinsics.
 	let extrinsics = extrinsics.into_iter().map(sign).collect::<Vec<_>>();

diff --git a/bin/node/runtime/src/lib.rs b/bin/node/runtime/src/lib.rs
@@ -126,6 +126,7 @@ pub const VERSION: RuntimeVersion = RuntimeVersion {
 	impl_version: 0,
 	apis: RUNTIME_API_VERSIONS,
 	transaction_version: 2,
+	state_version: 1,
 };
 
 /// The BABE epoch configuration at genesis.

diff --git a/client/api/src/backend.rs b/client/api/src/backend.rs
@@ -29,7 +29,7 @@ use sp_core::offchain::OffchainStorage;
 use sp_runtime::{
 	generic::BlockId,
 	traits::{Block as BlockT, HashFor, NumberFor},
-	Justification, Justifications, Storage,
+	Justification, Justifications, StateVersion, Storage,
 };
 use sp_state_machine::{
 	ChildStorageCollection, IndexOperation, OffchainChangesCollection, StorageCollection,
@@ -166,10 +166,15 @@ pub trait BlockImportOperation<Block: BlockT> {
 		&mut self,
 		storage: Storage,
 		commit: bool,
+		state_version: StateVersion,
 	) -> sp_blockchain::Result<Block::Hash>;
 
 	/// Inject storage data into the database replacing any existing data.
-	fn reset_storage(&mut self, storage: Storage) -> sp_blockchain::Result<Block::Hash>;
+	fn reset_storage(
+		&mut self,
+		storage: Storage,
+		state_version: StateVersion,
+	) -> sp_blockchain::Result<Block::Hash>;
 
 	/// Set storage changes.
 	fn update_storage(

diff --git a/client/api/src/call_executor.rs b/client/api/src/call_executor.rs
@@ -19,7 +19,7 @@
 //! A method call executor interface.
 
 use codec::{Decode, Encode};
-use sc_executor::RuntimeVersion;
+use sc_executor::{RuntimeVersion, RuntimeVersionOf};
 use sp_core::NativeOrEncoded;
 use sp_externalities::Extensions;
 use sp_runtime::{generic::BlockId, traits::Block as BlockT};
@@ -42,7 +42,7 @@ pub trait ExecutorProvider<Block: BlockT> {
 }
 
 /// Method call executor.
-pub trait CallExecutor<B: BlockT> {
+pub trait CallExecutor<B: BlockT>: RuntimeVersionOf {
 	/// Externalities error type.
 	type Error: sp_state_machine::Error;
 

diff --git a/client/api/src/in_mem.rs b/client/api/src/in_mem.rs
@@ -26,7 +26,7 @@ use sp_core::{
 use sp_runtime::{
 	generic::BlockId,
 	traits::{Block as BlockT, HashFor, Header as HeaderT, NumberFor, Zero},
-	Justification, Justifications, Storage,
+	Justification, Justifications, StateVersion, Storage,
 };
 use sp_state_machine::{
 	Backend as StateBackend, ChildStorageCollection, InMemoryBackend, IndexOperation,
@@ -506,6 +506,7 @@ where
 		&mut self,
 		storage: Storage,
 		commit: bool,
+		state_version: StateVersion,
 	) -> sp_blockchain::Result<Block::Hash> {
 		check_genesis_storage(&storage)?;
 
@@ -519,6 +520,7 @@ where
 		let (root, transaction) = self.old_state.full_storage_root(
 			storage.top.iter().map(|(k, v)| (k.as_ref(), Some(v.as_ref()))),
 			child_delta,
+			state_version,
 		);
 
 		if commit {
@@ -566,12 +568,17 @@ where
 		&mut self,
 		storage: Storage,
 		commit: bool,
+		state_version: StateVersion,
 	) -> sp_blockchain::Result<Block::Hash> {
-		self.apply_storage(storage, commit)
+		self.apply_storage(storage, commit, state_version)
 	}
 
-	fn reset_storage(&mut self, storage: Storage) -> sp_blockchain::Result<Block::Hash> {
-		self.apply_storage(storage, true)
+	fn reset_storage(
+		&mut self,
+		storage: Storage,
+		state_version: StateVersion,
+	) -> sp_blockchain::Result<Block::Hash> {
+		self.apply_storage(storage, true, state_version)
 	}
 
 	fn insert_aux<I>(&mut self, ops: I) -> sp_blockchain::Result<()>

diff --git a/client/block-builder/src/lib.rs b/client/block-builder/src/lib.rs
@@ -232,6 +232,7 @@ where
 			header.extrinsics_root().clone(),
 			HashFor::<Block>::ordered_trie_root(
 				self.extrinsics.iter().map(Encode::encode).collect(),
+				sp_runtime::StateVersion::V0,
 			),
 		);
 

diff --git a/client/db/src/bench.rs b/client/db/src/bench.rs
@@ -34,7 +34,7 @@ use sp_core::{
 };
 use sp_runtime::{
 	traits::{Block as BlockT, HashFor},
-	Storage,
+	StateVersion, Storage,
 };
 use sp_state_machine::{
 	backend::Backend as StateBackend, ChildStorageCollection, DBValue, ProofRecorder,
@@ -73,6 +73,7 @@ impl<Block: BlockT> sp_state_machine::Storage<HashFor<Block>> for StorageDb<Bloc
 		}
 	}
 }
+
 /// State that manages the backend database reference. Allows runtime to control the database.
 pub struct BenchmarkingState<B: BlockT> {
 	root: Cell<B::Hash>,
@@ -105,9 +106,10 @@ impl<B: BlockT> BenchmarkingState<B> {
 		record_proof: bool,
 		enable_tracking: bool,
 	) -> Result<Self, String> {
+		let state_version = sp_runtime::StateVersion::default();
 		let mut root = B::Hash::default();
 		let mut mdb = MemoryDB::<HashFor<B>>::default();
-		sp_state_machine::TrieDBMut::<HashFor<B>>::new(&mut mdb, &mut root);
+		sp_state_machine::TrieDBMutV1::<HashFor<B>>::new(&mut mdb, &mut root);
 
 		let mut state = BenchmarkingState {
 			state: RefCell::new(None),
@@ -138,6 +140,7 @@ impl<B: BlockT> BenchmarkingState<B> {
 			state.state.borrow_mut().as_mut().unwrap().full_storage_root(
 				genesis.top.iter().map(|(k, v)| (k.as_ref(), Some(v.as_ref()))),
 				child_delta,
+				state_version,
 			);
 		state.genesis = transaction.clone().drain();
 		state.genesis_root = root.clone();
@@ -415,28 +418,30 @@ impl<B: BlockT> StateBackend<HashFor<B>> for BenchmarkingState<B> {
 	fn storage_root<'a>(
 		&self,
 		delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
+		state_version: StateVersion,
 	) -> (B::Hash, Self::Transaction)
 	where
 		B::Hash: Ord,
 	{
 		self.state
 			.borrow()
 			.as_ref()
-			.map_or(Default::default(), |s| s.storage_root(delta))
+			.map_or(Default::default(), |s| s.storage_root(delta, state_version))
 	}
 
 	fn child_storage_root<'a>(
 		&self,
 		child_info: &ChildInfo,
 		delta: impl Iterator<Item = (&'a [u8], Option<&'a [u8]>)>,
+		state_version: StateVersion,
 	) -> (B::Hash, bool, Self::Transaction)
 	where
 		B::Hash: Ord,
 	{
 		self.state
 			.borrow()
 			.as_ref()
-			.map_or(Default::default(), |s| s.child_storage_root(child_info, delta))
+			.map_or(Default::default(), |s| s.child_storage_root(child_info, delta, state_version))
 	}
 
 	fn pairs(&self) -> Vec<(Vec<u8>, Vec<u8>)> {