Skip to content

Commit

Permalink
Upgrade trie-db from 0.28.0 to 0.29.0 (#3982)
Browse files Browse the repository at this point in the history
# Description
- What does this PR do?
1. Upgrades `trie-db`'s version to the latest release. This release
includes, among others, an implementation of `DoubleEndedIterator` for
the `TrieDB` struct, allowing to iterate both backwards and forwards
within the leaves of a trie.
2. Upgrades `trie-bench` to `0.39.0` for compatibility.
3. Upgrades `criterion` to `0.5.1` for compatibility.
- Why are these changes needed?
Besides keeping up with the upgrade of `trie-db`, this specifically adds
the functionality of iterating back on the leafs of a trie, with
`sp-trie`. In a project we're currently working on, this comes very
handy to verify a Merkle proof that is the response to a challenge. The
challenge is a random hash that (most likely) will not be an existing
leaf in the trie. So the challenged user, has to provide a Merkle proof
of the previous and next existing leafs in the trie, that surround the
random challenged hash.

Without having DoubleEnded iterators, we're forced to iterate until we
find the first existing leaf, like so:
```rust
        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        println!("RECONSTRUCTED TRIE {:#?}", trie);

        // Create an iterator over the leaf nodes.
        let mut iter = trie.iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        let mut prev_key = None;
        for element in &mut iter {
            if element.is_ok() {
                let (key, _) = element.unwrap();
                prev_key = Some(key);
                break;
            }
        }
        assert!(prev_key.is_some());

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        assert!(prev_key.unwrap() <= challenge_hash.to_vec());

        // The next element should exist (meaning there is no other existing leaf between the
        // previous and next leaf) and it should be greater than the challenged hash.
        let next_key = iter.next().unwrap().unwrap().0;
        assert!(next_key >= challenge_hash.to_vec());
```

With DoubleEnded iterators, we can avoid that, like this:
```rust
        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        // println!("RECONSTRUCTED TRIE {:#?}", trie);
        println!("\nChallenged key: {:?}", challenge_hash);

        // Create an iterator over the leaf nodes.
        let mut double_ended_iter = trie.into_double_ended_iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
        let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
        let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        println!("Prev key: {:?}", prev_key);
        assert!(prev_key <= challenge_hash.to_vec());

        println!("Next key: {:?}", next_key);
        assert!(next_key >= challenge_hash.to_vec());
```
- How were these changes implemented and what do they affect?
All that is needed for this functionality to be exposed is changing the
version number of `trie-db` in all the `Cargo.toml`s applicable, and
re-exporting some additional structs from `trie-db` in `sp-trie`.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
  • Loading branch information
ffarall and bkchr authored Apr 9, 2024
1 parent b6231c7 commit 4e73c0f
Show file tree
Hide file tree
Showing 8 changed files with 17 additions and 16 deletions.
13 changes: 6 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion bridges/primitives/runtime/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ sp-runtime = { path = "../../../substrate/primitives/runtime", default-features
sp-state-machine = { path = "../../../substrate/primitives/state-machine", default-features = false }
sp-std = { path = "../../../substrate/primitives/std", default-features = false }
sp-trie = { path = "../../../substrate/primitives/trie", default-features = false }
trie-db = { version = "0.28.0", default-features = false }
trie-db = { version = "0.29.0", default-features = false }

[dev-dependencies]
hex-literal = "0.4"
Expand Down
2 changes: 1 addition & 1 deletion cumulus/pallets/parachain-system/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ codec = { package = "parity-scale-codec", version = "3.0.0", default-features =
environmental = { version = "1.1.4", default-features = false }
impl-trait-for-tuples = "0.2.1"
log = { workspace = true }
trie-db = { version = "0.28.0", default-features = false }
trie-db = { version = "0.29.0", default-features = false }
scale-info = { version = "2.11.1", default-features = false, features = ["derive"] }

# Substrate
Expand Down
2 changes: 1 addition & 1 deletion substrate/primitives/state-machine/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ sp-core = { path = "../core", default-features = false }
sp-externalities = { path = "../externalities", default-features = false }
sp-panic-handler = { path = "../panic-handler", optional = true }
sp-trie = { path = "../trie", default-features = false }
trie-db = { version = "0.28.0", default-features = false }
trie-db = { version = "0.29.0", default-features = false }

[dev-dependencies]
array-bytes = "6.1"
Expand Down
6 changes: 3 additions & 3 deletions substrate/primitives/trie/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -32,16 +32,16 @@ rand = { version = "0.8", optional = true }
scale-info = { version = "2.11.1", default-features = false, features = ["derive"] }
thiserror = { optional = true, workspace = true }
tracing = { version = "0.1.29", optional = true }
trie-db = { version = "0.28.0", default-features = false }
trie-db = { version = "0.29.0", default-features = false }
trie-root = { version = "0.18.0", default-features = false }
sp-core = { path = "../core", default-features = false }
sp-externalities = { path = "../externalities", default-features = false }
schnellru = { version = "0.2.1", optional = true }

[dev-dependencies]
array-bytes = "6.1"
criterion = "0.4.0"
trie-bench = "0.38.0"
criterion = "0.5.1"
trie-bench = "0.39.0"
trie-standardmap = "0.16.0"
sp-runtime = { path = "../runtime" }

Expand Down
4 changes: 3 additions & 1 deletion substrate/primitives/trie/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,10 @@ use trie_db::proof::{generate_proof, verify_proof};
pub use trie_db::{
nibble_ops,
node::{NodePlan, ValuePlan},
triedb::{TrieDBDoubleEndedIterator, TrieDBKeyDoubleEndedIterator},
CError, DBValue, Query, Recorder, Trie, TrieCache, TrieConfiguration, TrieDBIterator,
TrieDBKeyIterator, TrieDBRawIterator, TrieLayout, TrieMut, TrieRecorder,
TrieDBKeyIterator, TrieDBNodeDoubleEndedIterator, TrieDBRawIterator, TrieLayout, TrieMut,
TrieRecorder,
};
pub use trie_db::{proof::VerifyError, MerkleValue};
/// The Substrate format implementation of `TrieStream`.
Expand Down
2 changes: 1 addition & 1 deletion substrate/test-utils/runtime/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ pallet-timestamp = { path = "../../frame/timestamp", default-features = false }
sp-consensus-grandpa = { path = "../../primitives/consensus/grandpa", default-features = false, features = ["serde"] }
sp-trie = { path = "../../primitives/trie", default-features = false }
sp-transaction-pool = { path = "../../primitives/transaction-pool", default-features = false }
trie-db = { version = "0.28.0", default-features = false }
trie-db = { version = "0.29.0", default-features = false }
sc-service = { path = "../../client/service", default-features = false, features = ["test-helpers"], optional = true }
sp-state-machine = { path = "../../primitives/state-machine", default-features = false }
sp-externalities = { path = "../../primitives/externalities", default-features = false }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ serde = { features = ["derive"], workspace = true, default-features = true }
sp-core = { path = "../../../../primitives/core" }
sp-state-machine = { path = "../../../../primitives/state-machine" }
sp-trie = { path = "../../../../primitives/trie" }
trie-db = "0.28.0"
trie-db = "0.29.0"

jsonrpsee = { version = "0.22", features = ["client-core", "macros", "server"] }

Expand Down

0 comments on commit 4e73c0f

Please sign in to comment.