From 2fbe8037faf76fdf3864899e3bfc239e370ae733 Mon Sep 17 00:00:00 2001 From: Slesarev Date: Wed, 8 Nov 2023 23:22:28 +0200 Subject: [PATCH 01/37] feat: first draft --- text/0000-metadata-for-offline-signers.md | 137 ++++++++++++++++++++++ 1 file changed, 137 insertions(+) create mode 100644 text/0000-metadata-for-offline-signers.md diff --git a/text/0000-metadata-for-offline-signers.md b/text/0000-metadata-for-offline-signers.md new file mode 100644 index 000000000..d9162af45 --- /dev/null +++ b/text/0000-metadata-for-offline-signers.md @@ -0,0 +1,137 @@ +# RFC-0000: Metadata for offline signers + +| | | +| --------------- | ------------------------------------------------------------------------------------------- | +| **Start Date** | 2023-10-31 | +| **Description** | Add SignedExtension to check Metadata Root Hash | +| **Authors** | Alzymologist Oy, Zondax LLC, Parity GmbH | + +## Summary + +Add a metadata digest value (33-byte constant within fixed `spec_version`) to Signed Extensions to supplement signer party with proof of correct extrinsic interpretation. The digest value is generated once before release and is well-known and deterministic. The digest mechanism is designed to be modular and flexible and to support partial metadata transfer as needed by the signing party's extrinsic decoding mechanism and taking into account signing devices potentially limited communication bandwidth and memory capacity. + +## Motivation + +### Background + +While all blockchain systems support (at least in some sense) offline signing used in air-gapped wallets and lightweight embedded devices, only few allow simultaneously complex upgradeable logic and full message decoding on the cold off-line signer side; Substrate is one of these heartening few, and therefore - we should build on this feature to greatly improve transaction security, and thus in general, network resilience. + +As a starting point, it is important to recognise that prudence and due care are naturally required. As we build further reliance on this feature we should be very careful to make sure it works correctly every time so as not to create false sense of security. + +In order to enable decoding that is small and optimized for chain storage transactions, a metadata entity is used, which is not at all small in itself (on the order of half-MB for most networks). This is a dynamic data chunk which completely describes chain interfaces and properties that could be made into a portable scale-encoded string for any given network version and passed along into an off-chain device to familiarize it with latest network updates. Of course, compromising this metadata anywhere in the path could result in differences between what user sees and signs, thus it is essential that we protect it. + +Therefore, we have 2 problems to be solved: + +1. Metadata is large, takes long time to be passed into a cold storage device with memory insufficient for its storage; metadata SHOULD be shortened and transmission SHOULD be optimized. +2. Metadata authenticity SHOULD be ensured. + +As of now, there is no working solution for (1), as the whole metadata has to be passed to the device. On top of this, the solution for (2) heavily relies on a trusted party managing keys and ensuring metadata is indeed authentic: creating poorly decentralized points of potential failure. + +### Solution requirements + +#### Include metadata digest into signature + +Some cryptographically strong digest of metadata MUST be included into signable blob. There SHALL NOT be storage overhead for this blob, nor computational overhead, on the node side; thus MUST be a constant within given runtime version, deterministically defined by metadata. + + - Metadata information that could be used in signable extrinsic decoding MUST be be included in digest; + - Digest MUST be deterministic with respect to metadata; + - Digest MUST be cryptographically strong against pre-image, both first and second; + - Extra-metadata information necessary for extrinsic decoding and constant within runtime version MUST be included in digest; + - Digest format SHOULD be versioned to allow rapid withdrawal of cold signing devices in case severe security vulnerability is found in shortener mechanism; + - Work necessary for proving metadata authenticity MAY be omitted at discretion of signer device design (to support automation tools). + +#### Reduce metadata size + +Metadata should be stripped from parts that are not necessary to parse a signable extrinsic, then it should be separated into a finite set of self-descriptive chunks. Thus, a subset of chunks necessary for signable extrinsic decoding and rendering could be sent, possibly in small portions (ultimately - one at a time), to cold device together with proof. + + - Single chunk with proof payload size SHOULD fit within few kB; + - Chunks handling mechanism SHOULD support chunks being sent in any order without memory utilization overhead; + - Unused enum variants MUST be stripped (this has great impact on transmitted metadata size; examples: era enum, enum with all calls for call batching). + +## Stakeholders + +This feature is essential for **all** offline signer tools; many regular signing tools might make use of it. In general, this RFC greatly improves security of any network implementing it, as many governing keys are used with offline signers. + +Implementing this RFC would remove requirement to maintain metadata portals manually, as task of metadata verification would be effectively moved to consensus mechanism of the chain. + +## Explanation + +Detailed description of metadata shortening and digest process is provided in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) crate (see `cargo doc --open` and examples). + +### Metadata descriptor + +Values for metadata shortening protocol version, `ExtrinsicMetadata`, SCALE-encoded `spec_version` and `spec_name` Strings, SCALE-encoded base58 prefix, SCALE-encoded decimals value, SCALE-encoded token unit String, should be prepared and combined as metadata descriptor. + +### Metadata modularization + +1. Types registry is stripped from `docs` fields. +2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants) + +### Digest + +1. Blake3 hash is computed for each chunk of modular short metadata registry. +2. Hashes are sorted and constructed into static Merkle tree as implemented in `merkle_cbt` crate using blake3 digest of concatenated child nodes values for merging. +3. Root hash of this tree is merged with metadata descriptor blake3 hash; this is metadata digest. + +Product of concatenation of porotocol version number with resulting metadata digest MUST be included into Signed Extensions + +### Shortening + +Shortened metadata is produced as sorted subset of modular metadata chunks selected from full set using protocol `cut_metadata` proposed in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) crate or equivalent operation that provides sufficient information for both decoding of extrinsic. + +### Transmission + +Shortened metadata chunks MAY be trasmitted into cold device together with Merkle proof in its entirety or in parts, depending on memory capabilities of the cold device and it ability to reconstruct larger fraction of tree. This document does not specify the manner of transmission. The order of metadata chunks MAY be arbitrary, the only requirement is that indices of nodes corresponding to chunks MUST be communicated. + +### Offline verification + +The transmitted metadata chunks are hashed together with proof lemmas to obtain root that MAY be transmitted along with the rest of payload. Verification that the root transmitted with message matches with calculated root is optional; the transmitted root SHOULD NOT be used in signature, calculated root MUST be used; however, there is no mechanism to enforce this - it should be done during cold signers code audit. + +### Chain verification + +The root of metadata computed by cold device MUST be included into Signed Extensions; this way the transaction will pass as valid iff hash of metadata as seen by cold storage device is identical to consensus hash of metadata, ensuring fair signing protocol. + +## Drawbacks + +### Increased transaction size + +Depending on implementation details, an extra byte may be needed to indicate whether the new version of metadata verification was used; this may be needed during transaction period, or the same byte may store the version of metadata hashing protocol + +### Transition overhead + +Some slightly out of spec systems might experience breaking changes as new content of signed extensions is added. It is important to note, that there is no real overhead in processing time nor complexity, as the metadata checking mechanism is voluntary. The only drawbacks are expected for tools that do not implement MetadataV14 self-descripting features. + +## Testing, Security, and Privacy + +The metadata shortening protocol should be extensively tested on all available examples of metadata before releasing changes to either metadata or shortener. Careful code review should be performed on shortener implementation code to ensure security. The main metadata tree would inevitably be constructed on runtime build which would also ensure correctness. + +To be able to recall shortener protocol in case of vulnerability issues, a version byte is included. + +## Performance, Ergonomics, and Compatibility + +### Performance + +This is negligibly short pessimization during build time on the chain side. Cold wallets performance would improve mostly as metadata validity mechanism that was taking most of effort in cold wallet support would become trivial. + +### Ergonomics + +The proposal was optimized for cold storage wallets usage with minimal impact on all other parts of the ecosystem + +### Compatibility + +Proposal in this form is not compatible with older toold that do not implement proper MetadataV14 self-descriptive features; those would have to be upgraded to include a new signed extensions field. + +## Prior Art and References + +This project was developed as Polkadot Treasury grant; relevant development links are located in [metadata-offline-project](https://github.com/Alzymologist/metadata-offline-project) repository. + +## Unresolved Questions + +1. Should hash inclusion bit be added to signed extensions? +2. How would polkadot-js handle the transition? +3. Where would non-rust tools like Ledger apps get shortened metadata content? + +## Future Directions and Related Material + +Changes to code of all cold signers to implement this mechanism SHOULD be done when this is enabled; non-cold signers may perform extra metadata check for better security. Ultimately, signing anything without decoding it with verifiable metadata should become discouraged in all situations where a decision-making mechanism is involved (that is, outside of fully automated blind signers like trade bots or staking rewards payout tools). + From 8d054453287071d8d4759ee282669b40c9f04846 Mon Sep 17 00:00:00 2001 From: Slesarev Date: Tue, 14 Nov 2023 23:16:28 +0200 Subject: [PATCH 02/37] docs: merkle tree construction rule --- text/0000-metadata-for-offline-signers.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/text/0000-metadata-for-offline-signers.md b/text/0000-metadata-for-offline-signers.md index d9162af45..c962181ff 100644 --- a/text/0000-metadata-for-offline-signers.md +++ b/text/0000-metadata-for-offline-signers.md @@ -56,7 +56,7 @@ Implementing this RFC would remove requirement to maintain metadata portals manu ## Explanation -Detailed description of metadata shortening and digest process is provided in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) crate (see `cargo doc --open` and examples). +Detailed description of metadata shortening and digest process is provided in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) crate (see `cargo doc --open` and examples). Below are presented algorithms of the process. ### Metadata descriptor @@ -67,13 +67,24 @@ Values for metadata shortening protocol version, `ExtrinsicMetadata`, SCALE-enco 1. Types registry is stripped from `docs` fields. 2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants) +### Complete Binary Merkle Tree construction protocol + +`blake3` transformation of concatenated child nodes (`blake3(left + right)`) as merge procedure; + +1. Leaves are numbered in ascending order. +2. Merge is performed using the leaf with highest index as right and node with second to highest index as left children; result is pushed to the end of nodes queue and leaves are discarded. +3. Step (2) is repeated until no leaves or just one leaf remains; in latter case, the last leaf is pushed to the front of the nodes queue. +4. Right node and then left node is popped from the front of the nodes queue and merged; the result is sent to the end of the queue. +5. Step (4) is repeated until only one node remains; this is tree root. + ### Digest 1. Blake3 hash is computed for each chunk of modular short metadata registry. -2. Hashes are sorted and constructed into static Merkle tree as implemented in `merkle_cbt` crate using blake3 digest of concatenated child nodes values for merging. -3. Root hash of this tree is merged with metadata descriptor blake3 hash; this is metadata digest. +2. Hashes are sorted in ascending order. +3. Complete Binary Merkle Tree is constructed as described above. +4. Root hash of this tree is merged with metadata descriptor blake3 hash; this is metadata digest. -Product of concatenation of porotocol version number with resulting metadata digest MUST be included into Signed Extensions +Product of concatenation of porotocol version number with resulting metadata digest MUST be included into Signed Extensions. ### Shortening From 36eb7173e5ca468789917fb6966cef81169decdc Mon Sep 17 00:00:00 2001 From: Slesarev Date: Wed, 15 Nov 2023 00:03:51 +0200 Subject: [PATCH 03/37] docs: specify minimal sufficient requirements for shortening --- text/0000-metadata-for-offline-signers.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/text/0000-metadata-for-offline-signers.md b/text/0000-metadata-for-offline-signers.md index c962181ff..b4926565f 100644 --- a/text/0000-metadata-for-offline-signers.md +++ b/text/0000-metadata-for-offline-signers.md @@ -65,13 +65,16 @@ Values for metadata shortening protocol version, `ExtrinsicMetadata`, SCALE-enco ### Metadata modularization 1. Types registry is stripped from `docs` fields. -2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants) +2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants). +3. Chunks are sorted by `id` in accending order; chunks with same `id` are sorted by enum vainant index in accending order. -### Complete Binary Merkle Tree construction protocol +### Merging protocol `blake3` transformation of concatenated child nodes (`blake3(left + right)`) as merge procedure; -1. Leaves are numbered in ascending order. +### Complete Binary Merkle Tree construction protocol + +1. Leaves are numbered in ascending order. Leaf index is associated with corresponding chunk. 2. Merge is performed using the leaf with highest index as right and node with second to highest index as left children; result is pushed to the end of nodes queue and leaves are discarded. 3. Step (2) is repeated until no leaves or just one leaf remains; in latter case, the last leaf is pushed to the front of the nodes queue. 4. Right node and then left node is popped from the front of the nodes queue and merged; the result is sent to the end of the queue. @@ -80,19 +83,18 @@ Values for metadata shortening protocol version, `ExtrinsicMetadata`, SCALE-enco ### Digest 1. Blake3 hash is computed for each chunk of modular short metadata registry. -2. Hashes are sorted in ascending order. 3. Complete Binary Merkle Tree is constructed as described above. -4. Root hash of this tree is merged with metadata descriptor blake3 hash; this is metadata digest. +4. Root hash of this tree (left) is merged with metadata descriptor blake3 hash (right); this is metadata digest. Product of concatenation of porotocol version number with resulting metadata digest MUST be included into Signed Extensions. ### Shortening -Shortened metadata is produced as sorted subset of modular metadata chunks selected from full set using protocol `cut_metadata` proposed in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) crate or equivalent operation that provides sufficient information for both decoding of extrinsic. +For shortening, an attempt to decode transaction completely using provided metadata is performed with the same algorithm that would be used on the cold side. All chunks are associated with their leaf indices. An example of this protocol is proposed in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) that is based on [substrate-parser](https://github.com/Alzymologist/substrate-parser) decoding protocol; any decoding protocol could be used here as long as cold signer's design finds it appropriate for given security model. ### Transmission -Shortened metadata chunks MAY be trasmitted into cold device together with Merkle proof in its entirety or in parts, depending on memory capabilities of the cold device and it ability to reconstruct larger fraction of tree. This document does not specify the manner of transmission. The order of metadata chunks MAY be arbitrary, the only requirement is that indices of nodes corresponding to chunks MUST be communicated. +Shortened metadata chunks MAY be trasmitted into cold device together with Merkle proof in its entirety or in parts, depending on memory capabilities of the cold device and it ability to reconstruct larger fraction of tree. This document does not specify the manner of transmission. The order of metadata chunks MAY be arbitrary, the only requirement is that indices of leaf nodes in Merkle tree corresponding to chunks MUST be communicated. ### Offline verification From acffee27902b38fae1b90dd6b0aeaaf1b45c40b3 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Wed, 15 Nov 2023 21:41:41 +0200 Subject: [PATCH 04/37] Update text/0000-metadata-for-offline-signers.md Co-authored-by: Carlo Sala --- text/0000-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-metadata-for-offline-signers.md b/text/0000-metadata-for-offline-signers.md index b4926565f..4424ce47b 100644 --- a/text/0000-metadata-for-offline-signers.md +++ b/text/0000-metadata-for-offline-signers.md @@ -86,7 +86,7 @@ Values for metadata shortening protocol version, `ExtrinsicMetadata`, SCALE-enco 3. Complete Binary Merkle Tree is constructed as described above. 4. Root hash of this tree (left) is merged with metadata descriptor blake3 hash (right); this is metadata digest. -Product of concatenation of porotocol version number with resulting metadata digest MUST be included into Signed Extensions. +Product of concatenation of protocol version number with resulting metadata digest MUST be included into Signed Extensions. ### Shortening From 0f800bf05a431765e43b056e60504cce0597e9d2 Mon Sep 17 00:00:00 2001 From: Slesarev Date: Mon, 20 Nov 2023 23:09:41 +0200 Subject: [PATCH 05/37] chore: set RFC index --- ...or-offline-signers.md => 0046-metadata-for-offline-signers.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-metadata-for-offline-signers.md => 0046-metadata-for-offline-signers.md} (100%) diff --git a/text/0000-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md similarity index 100% rename from text/0000-metadata-for-offline-signers.md rename to text/0046-metadata-for-offline-signers.md From 88704360c52345b992ad7b2daada05398a9a4d90 Mon Sep 17 00:00:00 2001 From: Slesarev Date: Wed, 22 Nov 2023 19:57:57 +0200 Subject: [PATCH 06/37] docs: apply suggestions from draft discussion --- text/0046-metadata-for-offline-signers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 4424ce47b..76acc07b2 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -60,12 +60,12 @@ Detailed description of metadata shortening and digest process is provided in [m ### Metadata descriptor -Values for metadata shortening protocol version, `ExtrinsicMetadata`, SCALE-encoded `spec_version` and `spec_name` Strings, SCALE-encoded base58 prefix, SCALE-encoded decimals value, SCALE-encoded token unit String, should be prepared and combined as metadata descriptor. +Values for metadata shortening protocol version, `ExtrinsicMetadata`, SCALE-encoded `spec_version` and `spec_name` Strings, `u16` base58 prefix, `u8` decimals value, SCALE-encoded token unit String, should be prepared and combined as metadata descriptor. ### Metadata modularization 1. Types registry is stripped from `docs` fields. -2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants). +2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants). Enums with 0 variants are treated as regular types. 3. Chunks are sorted by `id` in accending order; chunks with same `id` are sorted by enum vainant index in accending order. ### Merging protocol From 2c006b44d962e76f8ef86a1a59446dc0afc566b9 Mon Sep 17 00:00:00 2001 From: Slesarev Date: Tue, 5 Dec 2023 11:07:13 +0200 Subject: [PATCH 07/37] docs: more detailed description of signed extension and version byte --- text/0046-metadata-for-offline-signers.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 76acc07b2..b2a1144b3 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -31,9 +31,9 @@ As of now, there is no working solution for (1), as the whole metadata has to be #### Include metadata digest into signature -Some cryptographically strong digest of metadata MUST be included into signable blob. There SHALL NOT be storage overhead for this blob, nor computational overhead, on the node side; thus MUST be a constant within given runtime version, deterministically defined by metadata. +Some cryptographically strong digest of metadata MAY be included into signable blob. There SHALL NOT be storage overhead for this blob, nor computational overhead, on the node side; thus MUST be a constant within given runtime version, deterministically defined by metadata. - - Metadata information that could be used in signable extrinsic decoding MUST be be included in digest; + - Metadata information that could be used in signable extrinsic decoding MAY be be included in digest, its inclusion MUST be indicated in signed extensions; - Digest MUST be deterministic with respect to metadata; - Digest MUST be cryptographically strong against pre-image, both first and second; - Extra-metadata information necessary for extrinsic decoding and constant within runtime version MUST be included in digest; @@ -94,7 +94,7 @@ For shortening, an attempt to decode transaction completely using provided metad ### Transmission -Shortened metadata chunks MAY be trasmitted into cold device together with Merkle proof in its entirety or in parts, depending on memory capabilities of the cold device and it ability to reconstruct larger fraction of tree. This document does not specify the manner of transmission. The order of metadata chunks MAY be arbitrary, the only requirement is that indices of leaf nodes in Merkle tree corresponding to chunks MUST be communicated. +Shortened metadata chunks MAY be trasmitted into cold device together with Merkle proof in its entirety or in parts, depending on memory capabilities of the cold device and it ability to reconstruct larger fraction of tree. This document does not specify the manner of transmission. The order of metadata chunks MAY be arbitrary, the only requirement is that indices of leaf nodes in Merkle tree corresponding to chunks MUST be communicated. Community MAY handle proof format standartization independently. ### Offline verification @@ -102,13 +102,21 @@ The transmitted metadata chunks are hashed together with proof lemmas to obtain ### Chain verification -The root of metadata computed by cold device MUST be included into Signed Extensions; this way the transaction will pass as valid iff hash of metadata as seen by cold storage device is identical to consensus hash of metadata, ensuring fair signing protocol. +The root of metadata computed by cold device MAY be included into Signed Extensions; this way the transaction will pass as valid iff hash of metadata as seen by cold storage device is identical to consensus hash of metadata, ensuring fair signing protocol. + +The Signed Extension representing metadata digest is a single byte representing both digest vaule inclusion and shortening protocol version; this MUST be included in Signed Extensions set. Depending on its value, a digest value is included as `additionalSigned` to signature computation according to following specification: + +| signed extension value | digest value | comment | +|------------------------|----------------|------------------------------------| +| `0x00` | | digest is not included | +| `0x01` | 32-byte digest | this represents protocol version 1 | +| `0x02` - `0xFF` | *reserved* | reserved for future use | ## Drawbacks ### Increased transaction size -Depending on implementation details, an extra byte may be needed to indicate whether the new version of metadata verification was used; this may be needed during transaction period, or the same byte may store the version of metadata hashing protocol +A 1-byte increase in transaction size due to signed extension value. Digest is not included in transferred transaction, only in signing process. ### Transition overhead From 9b2beb2e929c4c652d6e8f247147179bf954453b Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 19 Dec 2023 12:13:57 +0200 Subject: [PATCH 08/37] Apply suggestions from code review --- text/0046-metadata-for-offline-signers.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index b2a1144b3..a7394abf4 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -8,7 +8,7 @@ ## Summary -Add a metadata digest value (33-byte constant within fixed `spec_version`) to Signed Extensions to supplement signer party with proof of correct extrinsic interpretation. The digest value is generated once before release and is well-known and deterministic. The digest mechanism is designed to be modular and flexible and to support partial metadata transfer as needed by the signing party's extrinsic decoding mechanism and taking into account signing devices potentially limited communication bandwidth and memory capacity. +Add a metadata digest value (33-byte constant within fixed `spec_version`) to Signed Extensions to supplement signer party with proof of correct extrinsic interpretation. The digest value is generated once before release and is well-known and deterministic. The digest mechanism is designed to be modular and flexible. It also supports partial metadata transfer as needed by the signing party's extrinsic decoding mechanism. This considers signing devices potentially limited communication bandwidth and/or memory capacity. ## Motivation @@ -37,7 +37,8 @@ Some cryptographically strong digest of metadata MAY be included into signable b - Digest MUST be deterministic with respect to metadata; - Digest MUST be cryptographically strong against pre-image, both first and second; - Extra-metadata information necessary for extrinsic decoding and constant within runtime version MUST be included in digest; - - Digest format SHOULD be versioned to allow rapid withdrawal of cold signing devices in case severe security vulnerability is found in shortener mechanism; + - It SHOULD be possible to quickly withdraw offline signing mechanism without access to cold signing devices; + - Digest format SHOULD be versioned. - Work necessary for proving metadata authenticity MAY be omitted at discretion of signer device design (to support automation tools). #### Reduce metadata size @@ -50,6 +51,8 @@ Metadata should be stripped from parts that are not necessary to parse a signabl ## Stakeholders +All chain teams are stakeholders, as implementing this feature would require timely effort on their side and would impact compatibility with older tools. + This feature is essential for **all** offline signer tools; many regular signing tools might make use of it. In general, this RFC greatly improves security of any network implementing it, as many governing keys are used with offline signers. Implementing this RFC would remove requirement to maintain metadata portals manually, as task of metadata verification would be effectively moved to consensus mechanism of the chain. @@ -60,7 +63,17 @@ Detailed description of metadata shortening and digest process is provided in [m ### Metadata descriptor -Values for metadata shortening protocol version, `ExtrinsicMetadata`, SCALE-encoded `spec_version` and `spec_name` Strings, `u16` base58 prefix, `u8` decimals value, SCALE-encoded token unit String, should be prepared and combined as metadata descriptor. +Values for: + +1. `u8` metadata shortening protocol version, +2. SCALE-encoded `ExtrinsicMetadata`, +3. SCALE-encoded `spec_version` `String` +4. SCALE-encoded `spec_name` String, +5. `u16` base58 prefix, +6. `u8` decimals value or `0u8` if no units are defined, +7. SCALE-encoded token unit String or empty string if no base units are defined, + +constitute metadata descriptor. This is minimal information sufficient to decode any signable transaction. Proof that the signing device was presented with genuine subset of Metadata Descriptor should be validated by chain. ### Metadata modularization @@ -148,7 +161,6 @@ This project was developed as Polkadot Treasury grant; relevant development link ## Unresolved Questions -1. Should hash inclusion bit be added to signed extensions? 2. How would polkadot-js handle the transition? 3. Where would non-rust tools like Ledger apps get shortened metadata content? From 888223a1cbdd3ebed0d7e0cf2233d98b9345a648 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Thu, 21 Dec 2023 11:59:26 +0200 Subject: [PATCH 09/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: Alexander Gryaznov --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index a7394abf4..44233e883 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -157,7 +157,7 @@ Proposal in this form is not compatible with older toold that do not implement p ## Prior Art and References -This project was developed as Polkadot Treasury grant; relevant development links are located in [metadata-offline-project](https://github.com/Alzymologist/metadata-offline-project) repository. +This project was developed upon a Polkadot Treasury grant; relevant development links are located in [metadata-offline-project](https://github.com/Alzymologist/metadata-offline-project) repository. ## Unresolved Questions From 0bb74b0255b10e499d31dc0ddc5be833feee2f6a Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Thu, 21 Dec 2023 11:59:48 +0200 Subject: [PATCH 10/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: Alexander Gryaznov --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 44233e883..3b91f25d5 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -153,7 +153,7 @@ The proposal was optimized for cold storage wallets usage with minimal impact on ### Compatibility -Proposal in this form is not compatible with older toold that do not implement proper MetadataV14 self-descriptive features; those would have to be upgraded to include a new signed extensions field. +Proposal in this form is not compatible with older tools that do not implement proper MetadataV14 self-descriptive features; those would have to be upgraded to include a new signed extensions field. ## Prior Art and References From c2708a8c8cafd05d27f54f38fde9657add14f3f4 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Thu, 21 Dec 2023 12:00:05 +0200 Subject: [PATCH 11/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: Alexander Gryaznov --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 3b91f25d5..69ec0ef7a 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -79,7 +79,7 @@ constitute metadata descriptor. This is minimal information sufficient to decode 1. Types registry is stripped from `docs` fields. 2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants). Enums with 0 variants are treated as regular types. -3. Chunks are sorted by `id` in accending order; chunks with same `id` are sorted by enum vainant index in accending order. +3. Chunks are sorted by `id` in ascending order; chunks with same `id` are sorted by enum variant index in ascending order. ### Merging protocol From 4f7d07bfe16ba485f3d78f012753cd506db629f7 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Thu, 21 Dec 2023 12:00:27 +0200 Subject: [PATCH 12/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: Alexander Gryaznov --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 69ec0ef7a..fa10e5289 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -33,7 +33,7 @@ As of now, there is no working solution for (1), as the whole metadata has to be Some cryptographically strong digest of metadata MAY be included into signable blob. There SHALL NOT be storage overhead for this blob, nor computational overhead, on the node side; thus MUST be a constant within given runtime version, deterministically defined by metadata. - - Metadata information that could be used in signable extrinsic decoding MAY be be included in digest, its inclusion MUST be indicated in signed extensions; + - Metadata information that could be used in signable extrinsic decoding MAY be included in digest, its inclusion MUST be indicated in signed extensions; - Digest MUST be deterministic with respect to metadata; - Digest MUST be cryptographically strong against pre-image, both first and second; - Extra-metadata information necessary for extrinsic decoding and constant within runtime version MUST be included in digest; From 439e119ef27a98d10c0211e8d7c582054a10e336 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 26 Dec 2023 22:15:35 +0200 Subject: [PATCH 13/37] More details and pseudocode examples (#1) * More details and pseudocode examples * Minor corrections --- text/0046-metadata-for-offline-signers.md | 116 ++++++++++++++++++++-- 1 file changed, 110 insertions(+), 6 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index fa10e5289..f647118d6 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -61,19 +61,83 @@ Implementing this RFC would remove requirement to maintain metadata portals manu Detailed description of metadata shortening and digest process is provided in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) crate (see `cargo doc --open` and examples). Below are presented algorithms of the process. -### Metadata descriptor +### Definitions + +#### Metadata structure + +Metadata in general consists of four sections: + +1. Types registry +2. Pallets +3. Extrinsic metadata +4. Runtime type + +Of these, only sections 1-3 contain information required for extrinsic decoding. The most important section is (1) Types registry, that is mostly used in extrinsic decoding. It is also the largest part, thus it is modularized for fractional transmission. Part (2) contains runtime version and is otherwise useless for transaction decoding; thus its contents are reduced to this parameter and included into Metadata Descriptor. Part (3) is included into Metadata Descriptor verbatim. + +#### Metadata descriptor Values for: 1. `u8` metadata shortening protocol version, 2. SCALE-encoded `ExtrinsicMetadata`, -3. SCALE-encoded `spec_version` `String` -4. SCALE-encoded `spec_name` String, +3. SCALE-encoded `spec_version` `String`, +4. SCALE-encoded `spec_name` `String`, 5. `u16` base58 prefix, 6. `u8` decimals value or `0u8` if no units are defined, -7. SCALE-encoded token unit String or empty string if no base units are defined, +7. SCALE-encoded `tokenSymbol` `String` defined on chain to identify the name of currency (available for example through `system.properties()` RPC call) or empty string if no base units are defined, + +``` +struct MetadataDescriptor { // really a scale-encoded enum, thus first field is enum value - only 0x01 currently supported. + protocol_version: u8, + extrinsic_metadata: Vec, // SCALE from `ExtrinsicMetadata + spec_version: Vec, // SCALE form `String` + spec_name: Vec, // SCALE from `String` + base58_prefix: u16, + decimals: u8, + token_symbol: Vec, // SCALE from `String` +} +``` + +constitute metadata descriptor. This is minimal information that is, together with (shortened) types registry, sufficient to decode any signable transaction. + +#### Merkle tree + +A **Complete Binary Merkle Tree** (**CBMT**) is proposed as digest structure. + +Every node of the proposed tree has a 32-bit value. + +A terminal node of the tree we call **leaf**. Its value is input for digest. + +The top node of the tree we call **root**. -constitute metadata descriptor. This is minimal information sufficient to decode any signable transaction. Proof that the signing device was presented with genuine subset of Metadata Descriptor should be validated by chain. +All node values for non-leave nodes are not terminal are computed through non-commutative **merge** procedure of child nodes. + +In CBMT, all layers must be populated, except for the last one, that must have complete filling from the left. + +Nodes are numbered top-down and left-to-right starting with 0 at the top of tree. + +``` +Example 8-node tree + + 0 + / \ + 1 2 + / \ / \ + 3 4 5 6 + / \ +7 8 + +Nodes 4, 5, 6, 7, 8 are leaves +Node 0 is root + +``` + +### General flow + +1. The metadata is converted into lean modular form (vector of chunks) +2. A Merkle tree is constructed from the metadata chunks +3. A root of tree (as a left element) is merged with Metadata Descriptor (as a right element) +4. Resulting value is a constant to be included in `additionalSigned` to prove that the metadata seen by cold device is genuine ### Metadata modularization @@ -81,6 +145,34 @@ constitute metadata descriptor. This is minimal information sufficient to decode 2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants). Enums with 0 variants are treated as regular types. 3. Chunks are sorted by `id` in ascending order; chunks with same `id` are sorted by enum variant index in ascending order. +``` +types_registry = metadataV14.types +modularized_registry = EmptyVector +for (id, type) in types.registry.iterate_enumerate { + type.doc = Null + if (type is ReduceableEnum) { // false for 0-variant enums + for variant in type.variants.iterate { + variant_type = Type { + path: type.path + type_params: Null + type_def: TypeDef::Variant(variants: [variant]) + } + modularized_registry.push(id, variant_type) + } + } else { + modularized_registry.push(id, type) + } +} + +modularized_registry.sort(|a, b| { + if a.id == b.id { //only possible for variants + a.variant_index > b.variant_index + } else { a.id > b.id } + } +) + +``` + ### Merging protocol `blake3` transformation of concatenated child nodes (`blake3(left + right)`) as merge procedure; @@ -93,13 +185,25 @@ constitute metadata descriptor. This is minimal information sufficient to decode 4. Right node and then left node is popped from the front of the nodes queue and merged; the result is sent to the end of the queue. 5. Step (4) is repeated until only one node remains; this is tree root. +``` +Resulting tree for metadata consisting of 5 nodes (numbered from 0 to 4): + + root + / \ + * * + / \ / \ + * 0 1 2 + / \ +3 4 +``` + ### Digest 1. Blake3 hash is computed for each chunk of modular short metadata registry. 3. Complete Binary Merkle Tree is constructed as described above. 4. Root hash of this tree (left) is merged with metadata descriptor blake3 hash (right); this is metadata digest. -Product of concatenation of protocol version number with resulting metadata digest MUST be included into Signed Extensions. +Version number and corresponding resulting metadata digest MUST be included into Signed Extensions as specified in Chain Verification section below. ### Shortening From ab1c0d40d6d95d1d01ff986f77bad156a23ba5d6 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Thu, 18 Jan 2024 12:42:22 +0200 Subject: [PATCH 14/37] Update text/0046-metadata-for-offline-signers.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Bastian Köcher --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index f647118d6..a0aed3bba 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -4,7 +4,7 @@ | --------------- | ------------------------------------------------------------------------------------------- | | **Start Date** | 2023-10-31 | | **Description** | Add SignedExtension to check Metadata Root Hash | -| **Authors** | Alzymologist Oy, Zondax LLC, Parity GmbH | +| **Authors** | Alzymologist Oy, Zondax LLC, Parity Technologies | ## Summary From 4c42ca9ffac54ce770ad4b321e0476efc40a7efe Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Thu, 18 Jan 2024 20:40:54 +0200 Subject: [PATCH 15/37] Update text/0046-metadata-for-offline-signers.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Bastian Köcher --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index a0aed3bba..b1aa2330c 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -31,7 +31,7 @@ As of now, there is no working solution for (1), as the whole metadata has to be #### Include metadata digest into signature -Some cryptographically strong digest of metadata MAY be included into signable blob. There SHALL NOT be storage overhead for this blob, nor computational overhead, on the node side; thus MUST be a constant within given runtime version, deterministically defined by metadata. +Some cryptographically strong digest of metadata MAY be included into signable blob. There SHALL NOT be storage overhead for this blob, nor computational overhead; thus MUST be a constant within a given runtime, deterministically defined by metadata. - Metadata information that could be used in signable extrinsic decoding MAY be included in digest, its inclusion MUST be indicated in signed extensions; - Digest MUST be deterministic with respect to metadata; From 613d1bb645446eea3828439b3a034eefacf530f4 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Thu, 18 Jan 2024 22:40:30 +0200 Subject: [PATCH 16/37] revert metadatadescriptor representation to enum --- text/0046-metadata-for-offline-signers.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index b1aa2330c..70314b764 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -87,9 +87,13 @@ Values for: 7. SCALE-encoded `tokenSymbol` `String` defined on chain to identify the name of currency (available for example through `system.properties()` RPC call) or empty string if no base units are defined, ``` -struct MetadataDescriptor { // really a scale-encoded enum, thus first field is enum value - only 0x01 currently supported. - protocol_version: u8, - extrinsic_metadata: Vec, // SCALE from `ExtrinsicMetadata +enum MetadataDescriptor { + V0, + V1(MetadataDescriptorV1), +} + +struct MetadataDescriptorV1 { + extrinsic_metadata: Vec, // SCALE from `ExtrinsicMetadata` spec_version: Vec, // SCALE form `String` spec_name: Vec, // SCALE from `String` base58_prefix: u16, From 674498d1c72c5876e8bb994f74e6f0f8eb8d6366 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Fri, 19 Jan 2024 10:14:07 +0200 Subject: [PATCH 17/37] replace nulls with empty vectors in pseudocode --- text/0046-metadata-for-offline-signers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 70314b764..ecb31a4a4 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -153,12 +153,12 @@ Node 0 is root types_registry = metadataV14.types modularized_registry = EmptyVector for (id, type) in types.registry.iterate_enumerate { - type.doc = Null + type.doc = empty_vector if (type is ReduceableEnum) { // false for 0-variant enums for variant in type.variants.iterate { variant_type = Type { path: type.path - type_params: Null + type_params: empty_vector type_def: TypeDef::Variant(variants: [variant]) } modularized_registry.push(id, variant_type) From 5ea8c9bc91207df0e1420d1fb3a53611f96ab1a5 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Fri, 19 Jan 2024 13:28:14 +0200 Subject: [PATCH 18/37] add pseudocode representation of type from `scale-info` --- text/0046-metadata-for-offline-signers.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index ecb31a4a4..8a64ea219 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -145,6 +145,22 @@ Node 0 is root ### Metadata modularization +Structure of types in shortened metadata exactly matches structure of types in `scale-info`, but `doc` field is always empty + +``` +struct Type { + path: Path, // vector of strings + type_params: Vec, + type_def: TypeDef, // enum of various types + doc: Vec, +} + +struct TypeParams { + name: String, + ty: Option, +} +``` + 1. Types registry is stripped from `docs` fields. 2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants). Enums with 0 variants are treated as regular types. 3. Chunks are sorted by `id` in ascending order; chunks with same `id` are sorted by enum variant index in ascending order. From 9c4ccb0dbc52e31339123fed1c72debbaa1a6530 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 23 Jan 2024 00:34:11 +0200 Subject: [PATCH 19/37] Update text/0046-metadata-for-offline-signers.md --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 8a64ea219..82c7b6489 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -239,7 +239,7 @@ The transmitted metadata chunks are hashed together with proof lemmas to obtain ### Chain verification -The root of metadata computed by cold device MAY be included into Signed Extensions; this way the transaction will pass as valid iff hash of metadata as seen by cold storage device is identical to consensus hash of metadata, ensuring fair signing protocol. +The root of metadata computed by cold device MAY be included into Signed Extensions; if it is included, the transaction will pass as valid iff hash of metadata as seen by cold storage device is identical to consensus hash of metadata, ensuring fair signing protocol. The Signed Extension representing metadata digest is a single byte representing both digest vaule inclusion and shortening protocol version; this MUST be included in Signed Extensions set. Depending on its value, a digest value is included as `additionalSigned` to signature computation according to following specification: From 94181782d863429b79bb8a155692afbc03e0453a Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 23 Jan 2024 00:55:23 +0200 Subject: [PATCH 20/37] Update text/0046-metadata-for-offline-signers.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Bastian Köcher --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 82c7b6489..ad946146d 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -140,7 +140,7 @@ Node 0 is root 1. The metadata is converted into lean modular form (vector of chunks) 2. A Merkle tree is constructed from the metadata chunks -3. A root of tree (as a left element) is merged with Metadata Descriptor (as a right element) +3. A root of tree is merged with the hash of the `MetadataDescriptor` 4. Resulting value is a constant to be included in `additionalSigned` to prove that the metadata seen by cold device is genuine ### Metadata modularization From 4f317c5c10da2f5a8af29003c140b4c8055d5d75 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 23 Jan 2024 01:07:33 +0200 Subject: [PATCH 21/37] pseudocode for tree construction --- text/0046-metadata-for-offline-signers.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index ad946146d..5facb81d4 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -205,6 +205,29 @@ modularized_registry.sort(|a, b| { 4. Right node and then left node is popped from the front of the nodes queue and merged; the result is sent to the end of the queue. 5. Step (4) is repeated until only one node remains; this is tree root. + +``` +queue = empty_queue + +while (leaves.length>1) { + right = leaves.pop_last + left = leaves.pop_last + queue.push_back(merge(left, right)) +} + +if leaves.length == 1 { + queue.push_front(leaves.last) +} + +while queue.len() > 1 { + right = queue.pop_front + left = queue.pop_front + queue.push_back(merge(left, right)) +} + +return queue.pop +``` + ``` Resulting tree for metadata consisting of 5 nodes (numbered from 0 to 4): From 7b472d10ccc19b3052fc159f540ab8e3e3537b26 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 23 Jan 2024 01:10:19 +0200 Subject: [PATCH 22/37] Remove shortening, transmission, and cold verification --- text/0046-metadata-for-offline-signers.md | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 5facb81d4..21a6f38aa 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -248,18 +248,6 @@ Resulting tree for metadata consisting of 5 nodes (numbered from 0 to 4): Version number and corresponding resulting metadata digest MUST be included into Signed Extensions as specified in Chain Verification section below. -### Shortening - -For shortening, an attempt to decode transaction completely using provided metadata is performed with the same algorithm that would be used on the cold side. All chunks are associated with their leaf indices. An example of this protocol is proposed in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) that is based on [substrate-parser](https://github.com/Alzymologist/substrate-parser) decoding protocol; any decoding protocol could be used here as long as cold signer's design finds it appropriate for given security model. - -### Transmission - -Shortened metadata chunks MAY be trasmitted into cold device together with Merkle proof in its entirety or in parts, depending on memory capabilities of the cold device and it ability to reconstruct larger fraction of tree. This document does not specify the manner of transmission. The order of metadata chunks MAY be arbitrary, the only requirement is that indices of leaf nodes in Merkle tree corresponding to chunks MUST be communicated. Community MAY handle proof format standartization independently. - -### Offline verification - -The transmitted metadata chunks are hashed together with proof lemmas to obtain root that MAY be transmitted along with the rest of payload. Verification that the root transmitted with message matches with calculated root is optional; the transmitted root SHOULD NOT be used in signature, calculated root MUST be used; however, there is no mechanism to enforce this - it should be done during cold signers code audit. - ### Chain verification The root of metadata computed by cold device MAY be included into Signed Extensions; if it is included, the transaction will pass as valid iff hash of metadata as seen by cold storage device is identical to consensus hash of metadata, ensuring fair signing protocol. From 4763f2a1ac9f99dada0be4a249191eb44bf9738f Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 23 Jan 2024 01:13:01 +0200 Subject: [PATCH 23/37] mention that type structure matcher particular metadata version --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 21a6f38aa..39273d2e7 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -145,7 +145,7 @@ Node 0 is root ### Metadata modularization -Structure of types in shortened metadata exactly matches structure of types in `scale-info`, but `doc` field is always empty +Structure of types in shortened metadata exactly matches structure of types in `scale-info` at MetadataV14 state, but `doc` field is always empty ``` struct Type { From b192d19a42bedc3026cbf2a9c587255b65039b33 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 00:00:14 +0200 Subject: [PATCH 24/37] Update text/0046-metadata-for-offline-signers.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Bastian Köcher --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 39273d2e7..12b72c81f 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -200,7 +200,7 @@ modularized_registry.sort(|a, b| { ### Complete Binary Merkle Tree construction protocol 1. Leaves are numbered in ascending order. Leaf index is associated with corresponding chunk. -2. Merge is performed using the leaf with highest index as right and node with second to highest index as left children; result is pushed to the end of nodes queue and leaves are discarded. +2. Merge is performed using the leaf with highest index as right and leaf with second to highest index as left children; result is pushed to the end of nodes queue and leaves are discarded. 3. Step (2) is repeated until no leaves or just one leaf remains; in latter case, the last leaf is pushed to the front of the nodes queue. 4. Right node and then left node is popped from the front of the nodes queue and merged; the result is sent to the end of the queue. 5. Step (4) is repeated until only one node remains; this is tree root. From 958e934ec095be26dbfd2f612b62da44315b0780 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 00:00:50 +0200 Subject: [PATCH 25/37] Update text/0046-metadata-for-offline-signers.md --- text/0046-metadata-for-offline-signers.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 12b72c81f..4662cc8d4 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -297,7 +297,6 @@ This project was developed upon a Polkadot Treasury grant; relevant development ## Unresolved Questions 2. How would polkadot-js handle the transition? -3. Where would non-rust tools like Ledger apps get shortened metadata content? ## Future Directions and Related Material From a2d87917f9e75e70e883985a3b4c890af010589c Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 00:01:25 +0200 Subject: [PATCH 26/37] Update text/0046-metadata-for-offline-signers.md --- text/0046-metadata-for-offline-signers.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 4662cc8d4..6ce7310cf 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -296,7 +296,6 @@ This project was developed upon a Polkadot Treasury grant; relevant development ## Unresolved Questions -2. How would polkadot-js handle the transition? ## Future Directions and Related Material From 6d9603b7fc52e0aad84852c2b9840d930373fde6 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 00:02:08 +0200 Subject: [PATCH 27/37] Update text/0046-metadata-for-offline-signers.md --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 6ce7310cf..2a5a92ebe 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -268,7 +268,7 @@ A 1-byte increase in transaction size due to signed extension value. Digest is n ### Transition overhead -Some slightly out of spec systems might experience breaking changes as new content of signed extensions is added. It is important to note, that there is no real overhead in processing time nor complexity, as the metadata checking mechanism is voluntary. The only drawbacks are expected for tools that do not implement MetadataV14 self-descripting features. +Some slightly out of spec systems might experience breaking changes as new content of signed extensions is added - tools that delay their transition instead of preparing ahead of time would break for the duration of delay. It is important to note, that there is no real overhead in processing time nor complexity, as the metadata checking mechanism is voluntary. ## Testing, Security, and Privacy From 10bfa005aab3ec6725081eb93735f24a6c5a83c1 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 00:02:25 +0200 Subject: [PATCH 28/37] Update text/0046-metadata-for-offline-signers.md --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 2a5a92ebe..e25ae5e04 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -35,7 +35,7 @@ Some cryptographically strong digest of metadata MAY be included into signable b - Metadata information that could be used in signable extrinsic decoding MAY be included in digest, its inclusion MUST be indicated in signed extensions; - Digest MUST be deterministic with respect to metadata; - - Digest MUST be cryptographically strong against pre-image, both first and second; + - Digest MUST be cryptographically strong against pre-image, both first (finding an input that results in given digest) and second (finding an input that results in same digest as some other input given); - Extra-metadata information necessary for extrinsic decoding and constant within runtime version MUST be included in digest; - It SHOULD be possible to quickly withdraw offline signing mechanism without access to cold signing devices; - Digest format SHOULD be versioned. From 17b046caa066e1fec40710ff523762b8298e6b27 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 00:02:51 +0200 Subject: [PATCH 29/37] Update text/0046-metadata-for-offline-signers.md --- text/0046-metadata-for-offline-signers.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index e25ae5e04..4b49f743a 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -8,7 +8,9 @@ ## Summary -Add a metadata digest value (33-byte constant within fixed `spec_version`) to Signed Extensions to supplement signer party with proof of correct extrinsic interpretation. The digest value is generated once before release and is well-known and deterministic. The digest mechanism is designed to be modular and flexible. It also supports partial metadata transfer as needed by the signing party's extrinsic decoding mechanism. This considers signing devices potentially limited communication bandwidth and/or memory capacity. +Add a metadata digest value to signed data to supplement signer party with proof of correct extrinsic interpretation. This would ensure that hardware wallets always use correct metadata to decode the information for the user. + +The digest value is generated once before release and is well-known and deterministic. The digest mechanism is designed to be modular and flexible. It also supports partial metadata transfer as needed by the signing party's extrinsic decoding mechanism. This considers signing devices potentially limited communication bandwidth and/or memory capacity. ## Motivation From 21dc94aa042ce7ff17e18f6e1273e33b269f28a5 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 00:03:32 +0200 Subject: [PATCH 30/37] Update text/0046-metadata-for-offline-signers.md --- text/0046-metadata-for-offline-signers.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 4b49f743a..834d8bfcd 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -65,16 +65,6 @@ Detailed description of metadata shortening and digest process is provided in [m ### Definitions -#### Metadata structure - -Metadata in general consists of four sections: - -1. Types registry -2. Pallets -3. Extrinsic metadata -4. Runtime type - -Of these, only sections 1-3 contain information required for extrinsic decoding. The most important section is (1) Types registry, that is mostly used in extrinsic decoding. It is also the largest part, thus it is modularized for fractional transmission. Part (2) contains runtime version and is otherwise useless for transaction decoding; thus its contents are reduced to this parameter and included into Metadata Descriptor. Part (3) is included into Metadata Descriptor verbatim. #### Metadata descriptor From 78b8e67a3847bd578a7c87b7d3ce3480b160ba1a Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 18:46:35 +0200 Subject: [PATCH 31/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: Xiliang Chen --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 834d8bfcd..5f7a369ab 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -74,7 +74,7 @@ Values for: 2. SCALE-encoded `ExtrinsicMetadata`, 3. SCALE-encoded `spec_version` `String`, 4. SCALE-encoded `spec_name` `String`, -5. `u16` base58 prefix, +5. `u16` ss58 prefix, 6. `u8` decimals value or `0u8` if no units are defined, 7. SCALE-encoded `tokenSymbol` `String` defined on chain to identify the name of currency (available for example through `system.properties()` RPC call) or empty string if no base units are defined, From ebfa55f29e896c3c6ac54bc8e6a4124d6fa8aeb2 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 18:46:57 +0200 Subject: [PATCH 32/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: Xiliang Chen --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 5f7a369ab..34226f419 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -88,7 +88,7 @@ struct MetadataDescriptorV1 { extrinsic_metadata: Vec, // SCALE from `ExtrinsicMetadata` spec_version: Vec, // SCALE form `String` spec_name: Vec, // SCALE from `String` - base58_prefix: u16, + ss58_prefix: u16, decimals: u8, token_symbol: Vec, // SCALE from `String` } From a00ecf75ea0a4ad0222d8f976dacf17f0f8ed8cc Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 19:11:19 +0200 Subject: [PATCH 33/37] remove old mentions of scale encoding from metadata descriptor --- text/0046-metadata-for-offline-signers.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 34226f419..439b9d0a8 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -71,12 +71,12 @@ Detailed description of metadata shortening and digest process is provided in [m Values for: 1. `u8` metadata shortening protocol version, -2. SCALE-encoded `ExtrinsicMetadata`, -3. SCALE-encoded `spec_version` `String`, -4. SCALE-encoded `spec_name` `String`, +2. `ExtrinsicMetadata`, +3. `spec_version` `String`, +4. `spec_name` `String`, 5. `u16` ss58 prefix, 6. `u8` decimals value or `0u8` if no units are defined, -7. SCALE-encoded `tokenSymbol` `String` defined on chain to identify the name of currency (available for example through `system.properties()` RPC call) or empty string if no base units are defined, +7. `tokenSymbol` `String` defined on chain to identify the name of currency (available for example through `system.properties()` RPC call) or empty string if no base units are defined, ``` enum MetadataDescriptor { @@ -85,12 +85,12 @@ enum MetadataDescriptor { } struct MetadataDescriptorV1 { - extrinsic_metadata: Vec, // SCALE from `ExtrinsicMetadata` - spec_version: Vec, // SCALE form `String` - spec_name: Vec, // SCALE from `String` + extrinsic_metadata: ExtrinsicMetadata + spec_version: String + spec_name: String ss58_prefix: u16, decimals: u8, - token_symbol: Vec, // SCALE from `String` + token_symbol: String } ``` From 91e9bdf4a829897e9f2e7aa3e1c7126aad543993 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 20:55:47 +0200 Subject: [PATCH 34/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 439b9d0a8..8449b1975 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -1,4 +1,4 @@ -# RFC-0000: Metadata for offline signers +# RFC-0046: Metadata for Offline Signers | | | | --------------- | ------------------------------------------------------------------------------------------- | From 2de8cd71bc594eff07846b62ec9b31a80cb7df19 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Tue, 30 Jan 2024 21:16:27 +0200 Subject: [PATCH 35/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: Xiliang Chen --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 8449b1975..00bfffd15 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -201,7 +201,7 @@ modularized_registry.sort(|a, b| { ``` queue = empty_queue -while (leaves.length>1) { +while leaves.length > 1 { right = leaves.pop_last left = leaves.pop_last queue.push_back(merge(left, right)) From 6cb73814117465729d557b722f59c4d86654ccf1 Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Mon, 5 Feb 2024 21:43:08 +0200 Subject: [PATCH 36/37] docs: modified V15-oriented shortened structure and Basti's suggestions (#2) * docs: modified V15-oriented shortened structure * Include Basti's writings * Update text/0046-metadata-for-offline-signers.md Co-authored-by: Carlo Sala * Update text/0046-metadata-for-offline-signers.md Co-authored-by: Carlo Sala * Update text/0046-metadata-for-offline-signers.md * add string runtime version argument * Update text/0046-metadata-for-offline-signers.md Co-authored-by: Carlo Sala * Update text/0046-metadata-for-offline-signers.md Co-authored-by: Carlo Sala * Update text/0046-metadata-for-offline-signers.md Co-authored-by: Carlo Sala * remove examples of what `spec_version` could be in a ideomatic universe * docs: grammar --------- Co-authored-by: Carlo Sala --- text/0046-metadata-for-offline-signers.md | 168 ++++++++++++++++++---- 1 file changed, 139 insertions(+), 29 deletions(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 00bfffd15..4d23b5bc8 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -1,13 +1,17 @@ # RFC-0046: Metadata for Offline Signers -| | | -| --------------- | ------------------------------------------------------------------------------------------- | -| **Start Date** | 2023-10-31 | -| **Description** | Add SignedExtension to check Metadata Root Hash | -| **Authors** | Alzymologist Oy, Zondax LLC, Parity Technologies | +| | | +| --------------- | -------------------------------------------------------------------------------------------- | +| **Start Date** | 2023-10-31 | +| **Description** | Include merkelized metadata hash in extrinsic signature for trust-less metadata verification.| +| **Authors** | Alzymologist Oy, Zondax LLC, Parity Technologies | ## Summary +To interact with chains in the Polkadot ecosystem it is required to know how transactions are encoded and how to read state. For doing this Substrate, the framework used by most of the chains in the Polkadot ecosystem, exposes metadata about the runtime to the outside. UIs, wallets, signers, etc can use this metadata to interact with the chains. This makes the metadata a crucial piece of the transaction encoding as users are relying on the interacting software to encode the transactions in the correct format. It gets even more important when the user signs the transaction on an offline signer as the latter by its nature can not get access to the metadata without relying on the online wallet to provide it the metadata. So, the idea is that the metadata is chunked and these chunks are put into a merkle tree. The root hash of this merkle tree represents the metadata. Cryptographically digested together small number of extra-metadata information, it allows the offline signers and wallets to decode transactions by getting proofs for the individual chunks of the metadata. This digest is also included into the signed data of the transaction (but not sent as part of the transaction). The runtime is then including its known metadata root hash and extra-metadata constants when verifying the transaction. If the digest known by the runtime differs from the one that the offline signer used, it very likely means that the online wallet provided some invalid data and the verification of the transaction fails thus making metadata substitution impossible. The user is depending on the offline wallet showing them the correct decoded transaction before signing and with the merkelized metadata they can be sure that it was the correct transaction or the runtime will reject the transaction. + +Thus we propose following: + Add a metadata digest value to signed data to supplement signer party with proof of correct extrinsic interpretation. This would ensure that hardware wallets always use correct metadata to decode the information for the user. The digest value is generated once before release and is well-known and deterministic. The digest mechanism is designed to be modular and flexible. It also supports partial metadata transfer as needed by the signing party's extrinsic decoding mechanism. This considers signing devices potentially limited communication bandwidth and/or memory capacity. @@ -53,30 +57,38 @@ Metadata should be stripped from parts that are not necessary to parse a signabl ## Stakeholders +- Runtime implementors +- UI/wallet implementors +- Offline wallet implementors + All chain teams are stakeholders, as implementing this feature would require timely effort on their side and would impact compatibility with older tools. This feature is essential for **all** offline signer tools; many regular signing tools might make use of it. In general, this RFC greatly improves security of any network implementing it, as many governing keys are used with offline signers. Implementing this RFC would remove requirement to maintain metadata portals manually, as task of metadata verification would be effectively moved to consensus mechanism of the chain. +The idea for this RFC was brought up by runtime implementors and was extensively discussed with offline wallet implementors. It was designed in such a way that it can work easily with the existing offline wallet solutions in the Polkadot ecosystem. + ## Explanation -Detailed description of metadata shortening and digest process is provided in [metadata-shortener](https://github.com/Alzymologist/metadata-shortener) crate (see `cargo doc --open` and examples). Below are presented algorithms of the process. +The FRAME metadata provides a lot of information about a FRAME-based runtime. It contains information about the pallets, the calls per pallet, the storage entries per pallet, information about runtime APIs and type information about most of the types used in the runtime. For decoding transactions on an offline wallet, we mainly require this type of information. Most of the other information in the FRAME metadata is actually not required for the decoding, and thus, we can remove them. Thus, we have come up with a custom representation of the metadata and how this custom metadata is chunked, ensuring that we only need to send the chunks required for decoding a certain transaction to the offline wallet. -### Definitions +A reference implementation of this process is provided in [metadata-shortener](https://docs.rs/metadata-shortener/latest/metadata_shortener/) crate. +### Definitions #### Metadata descriptor Values for: -1. `u8` metadata shortening protocol version, -2. `ExtrinsicMetadata`, -3. `spec_version` `String`, -4. `spec_name` `String`, -5. `u16` ss58 prefix, -6. `u8` decimals value or `0u8` if no units are defined, -7. `tokenSymbol` `String` defined on chain to identify the name of currency (available for example through `system.properties()` RPC call) or empty string if no base units are defined, +1. `u8` metadata shortening protocol version (as encoded enum variant), +2. The `Id` of type of the outermost Call enum (u32) as described in https://docs.rs/frame-metadata/latest/frame_metadata/v15/struct.ExtrinsicMetadata.html#structfield.call_ty +3. Vector of `SignedExtension` structs as defined in Metadata V15 (vector of named (`String`) pairs of `TypeId` (`u32`)) +4. `spec_version` `String` (see explanation below) as found in the `RuntimeVersion` at generation of the metadata. While this information can also be found in the metadata, it is hidden in a big blob of data. To avoid transferring this big blob of data, we directly add these values here, +5. `spec_name` `String` as found in the `RuntimeVersion`, +6. `u16` ss58 prefix, +7. `u8` decimals value or `0u8` if no units are defined, +8. `tokenSymbol` `String` defined on chain to identify the name of currency (available for example through `system.properties()` RPC call) or empty `String` if no base units are defined, ``` enum MetadataDescriptor { @@ -85,17 +97,28 @@ enum MetadataDescriptor { } struct MetadataDescriptorV1 { - extrinsic_metadata: ExtrinsicMetadata - spec_version: String - spec_name: String + call_ty: u32, // TypeId + signed_extensions: Vec, + spec_version: String, + spec_name: String, ss58_prefix: u16, decimals: u8, token_symbol: String } + +struct SignedExtensionMetadata { + identifier: String, + ty: u32, // TypeId + additional_signed: u32, // TypeId +} ``` constitute metadata descriptor. This is minimal information that is, together with (shortened) types registry, sufficient to decode any signable transaction. +**Note on `spec_version`**: this type is described in metadata; it is indeed u32 in Polkadot and Kusama and most other networks and thus could be defined in a typesafe manner, but theoretically, it could be anything, as long as it is serializable. We saw a couple of networks using u16 there in the past. Not that it's all that important, but as long as our protocol would be the only limitation - it would be a limitation for our protocol; it would be unwise to enforce it here without first enforcing it upstream. + +The version is not exactly an algebraic number - it has comparison operation, but no other algebra is defined; the primitive variable with these properties is really a `String`, not an integer. We've argued a lot about this internally, but with a naive JS-style solution, we have all the flexibility we might want and no real downsides (as from the point of view of our protocol, it's just an opaque constant - key for database search). + #### Merkle tree A **Complete Binary Merkle Tree** (**CBMT**) is proposed as digest structure. @@ -137,9 +160,14 @@ Node 0 is root ### Metadata modularization -Structure of types in shortened metadata exactly matches structure of types in `scale-info` at MetadataV14 state, but `doc` field is always empty +Structure of types in shortened metadata exactly matches structure of types in `scale-info` at MetadataV15 state, but `doc` field is always empty. This type system provides sufficient information for decoding with SCALE. For those not familiar with MetadataV15, a brief excerpt of types structure is presented below; however, it is important, that the protocol implemented in this RFC MUST agree directly to `scale-info` definitions, to ensure single source of truth and resolve possible diversions towards more adoptable and universal solution. ``` +struct Chunk { + id: u32, + type: Type, +} + struct Type { path: Path, // vector of strings type_params: Vec, @@ -151,23 +179,100 @@ struct TypeParams { name: String, ty: Option, } + +enum TypeDef { + Composite(TypeDefComposite), + Variant(TypeDefVariant), + Sequence(TypeDefSequence), + Array(TypeDefArray), + Tuple(TypeDefTuple), + Primitive(TypeDefPrimitive), + Compact(TypeDefCompact), + BitSequence(TypeDefBitSequence), +} + +struct TypeDefComposite { + fields: Vec, +} + +pub struct Field { + name: Option, + ty: Type, + type_name: Option, + docs: Vec, +} + +struct TypeDefVariant { + variants: Vec, +} + +struct Variant { + name: String, + fields: Vec, + index: u8, + docs: Vec, +} + +struct TypeDefSequence { + type_param: Type, +} + +struct TypeDefArray { + len: u32, + type_param: Type, +} + +struct TypeDefTuple { + fields: Vec, +} + +enum TypeDefPrimitive { + bool, + char, + String, + u8, + u16, + u32, + u64, + u128, + u256, + i8, + i16, + i32, + i64, + i128, + i256, +} + +struct TypeDefCompact { + type_param: Type, +} + +struct TypeDefBitSequence { + pub bit_store_type: Type, + pub bit_order_type: Type, +} + ``` +Modularization happens thus: + 1. Types registry is stripped from `docs` fields. 2. Types records are separated into chunks, with enum variants being individual chunks differing by variant index; each chunk consisting of `id` (same as in full metadata registry) and SCALE-encoded 'Type' description (reduced to 1-variant enum for enum variants). Enums with 0 variants are treated as regular types. 3. Chunks are sorted by `id` in ascending order; chunks with same `id` are sorted by enum variant index in ascending order. ``` -types_registry = metadataV14.types -modularized_registry = EmptyVector +types_registry = metadataV15.types +modularized_registry = EmptyVector for (id, type) in types.registry.iterate_enumerate { type.doc = empty_vector if (type is ReduceableEnum) { // false for 0-variant enums for variant in type.variants.iterate { variant_type = Type { - path: type.path - type_params: empty_vector - type_def: TypeDef::Variant(variants: [variant]) + path: type.path, + type_params: empty_vector, + type_def: TypeDef::Variant(variants: [variant]), + docs: empty_vector, } modularized_registry.push(id, variant_type) } @@ -185,6 +290,9 @@ modularized_registry.sort(|a, b| { ``` +Chunks are scale-encoded and digested by `blake3` algorithm into 32-byte CBMT leaf values. + + ### Merging protocol `blake3` transformation of concatenated child nodes (`blake3(left + right)`) as merge procedure; @@ -264,7 +372,9 @@ Some slightly out of spec systems might experience breaking changes as new conte ## Testing, Security, and Privacy -The metadata shortening protocol should be extensively tested on all available examples of metadata before releasing changes to either metadata or shortener. Careful code review should be performed on shortener implementation code to ensure security. The main metadata tree would inevitably be constructed on runtime build which would also ensure correctness. +All implementations MUST follow the RFC to generate the metadata hash. This includes which hash function to use and how to construct the metadata types tree. So, all implementations are following the same security criteria. As the chains will calculate the metadata hash at compile time, the build process generally needs to be trusted. However, this is already a solved problem in the Polkadot ecosystem using reproducible builds. So, anyone can rebuild a chain runtime to ensure that a proposal is actually containing the changes as advertised. +Implementation can also be tested easily against each other by taking some metadata and ensuring that they all come to the same metadata hash. +Privacy of users should also not be impacted. This assumes that wallets will generate the metadata hash locally and don't leak any information to third party services about which chunks a user will send to its offline wallet. Besides that there is no leak of private information as getting the raw metadata from the chain is an operation that is done by almost everyone. To be able to recall shortener protocol in case of vulnerability issues, a version byte is included. @@ -274,18 +384,18 @@ To be able to recall shortener protocol in case of vulnerability issues, a versi This is negligibly short pessimization during build time on the chain side. Cold wallets performance would improve mostly as metadata validity mechanism that was taking most of effort in cold wallet support would become trivial. -### Ergonomics - -The proposal was optimized for cold storage wallets usage with minimal impact on all other parts of the ecosystem +There should be no measurable impact on performance to Polkadot or any other chain using this feature. The metadata root hash is calculated at compile time and at runtime it is optionally used when checking the signature of a transaction. This means that at runtime no performance heavy operations are done. -### Compatibility +### Ergonomics -Proposal in this form is not compatible with older tools that do not implement proper MetadataV14 self-descriptive features; those would have to be upgraded to include a new signed extensions field. +The proposal alters the way a transaction is build, signed and verified. So, this imposes some required changes to any kind of developer that wants to construct transactions for Polkadot or any chain using this feature. As the developer can pass 0 for disabling the verification of the metadata root hash, it can be easily ignored. ## Prior Art and References This project was developed upon a Polkadot Treasury grant; relevant development links are located in [metadata-offline-project](https://github.com/Alzymologist/metadata-offline-project) repository. +There doesn't seem to exist a similar solution to what the RFC proposes right now. However, there are other solutions to the problem of trusted signing. Cosmos for example has a standardized way of transforming a transaction into some textual representation and this textual representation is included in the signed data. Basically achieving the same as what the RFC proposes, but it requires that for every transaction applied in a block, every node in the network always has to generate this textual representation to ensure the transaction signature is valid. Furthermore, since rendering of this textual representation is potentially non-trivial (with use of non-printed glyphs, direction control, escape characters, etc.), inclusion of arbitrary textual representation increases attack surface and potentially results in subtly false sense of security. + ## Unresolved Questions From 601c623921ffeb2df885fa50e4d6845886f4cfaf Mon Sep 17 00:00:00 2001 From: Slesarew <33295157+Slesarew@users.noreply.github.com> Date: Wed, 21 Feb 2024 08:34:47 +0200 Subject: [PATCH 37/37] Update text/0046-metadata-for-offline-signers.md Co-authored-by: Juan Leni --- text/0046-metadata-for-offline-signers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0046-metadata-for-offline-signers.md b/text/0046-metadata-for-offline-signers.md index 4d23b5bc8..c0cf7df32 100644 --- a/text/0046-metadata-for-offline-signers.md +++ b/text/0046-metadata-for-offline-signers.md @@ -4,7 +4,7 @@ | --------------- | -------------------------------------------------------------------------------------------- | | **Start Date** | 2023-10-31 | | **Description** | Include merkelized metadata hash in extrinsic signature for trust-less metadata verification.| -| **Authors** | Alzymologist Oy, Zondax LLC, Parity Technologies | +| **Authors** | Alzymologist Oy, Zondax AG, Parity Technologies | ## Summary