Skip to content

Commit

Permalink
Metadata V15: Generate Runtime APIs (#918)
Browse files Browse the repository at this point in the history 
* Update frame-metadata to v15.1.0

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Enable V15 unstable metadata in frame-metadata

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Move validation hashing to dedicated file

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Use sp-metadata-ir from substrate to work with metadata

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Revert using sp-metadata-ir in favor of conversion to v15

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Convert v14 to v15

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Use v15 for validation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Use v15 for codegen

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata/bench: Use v15

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust to v15 metadata

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Adjust testing

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Improve documentation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* force CI

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc: Fetch metadata at version

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* artifacts: Update polkadot.scale from commit 6dc9e84dde2

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Fetch V15 using the new API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Add runtime API interface

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Hash runtime API metadata for validation

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Extract runtime API metadata wrapper from subxt::Metadata

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt: Adjust hashing cache to reflect root+item keys

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* rpc: Add raw state_call API method

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* runtime_api: Add payload with static and dynamic variants

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt: Allow payloads to call into the runtime

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* examples: Add example to make a runtime API call both static and dynamic

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update polkadot.rs

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Simplify client fetching

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Address feedback and fallback to old API if needed

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* runtime_api: Make mutability conditional on input params

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Regenerate polkadot.rs

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Retain only pallets without runtime API info

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Retry via `Metadata_metadata` without conversion

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* payload: Remove `Decode` and change validation fn

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* metadata: Retain runtime API types

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Runtime APIs documentation based on flag

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update examples/examples/custom_metadata_url.rs

Co-authored-by: James Wilson <james@jsdw.me>

* Update artifacts from polkadot-a6cfdb16e9

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update polkadot.rs with polkadot-a6cfdb16e9

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Generate input structures for runtime API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* runtime_api: Remove the static paylaod and use single impl

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* examples: Fetch account nonce

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Adjust build script to fetch latest metadata

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Check account nonce from runtime API

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update cargo.lock

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Fix doc generation for runtime types

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Rename `inputs` runtime calls module to `types`

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Generate Calls structs inside the types module

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Check Alice account nonce before submitting the tx

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* cli: Add metadata version option flag supporting v14 and unstable

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* cli: Specify version to fetch

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt: Fallback to fetching latest stable metadata

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* subxt: Add unstable-metadata feature to fetch the latest

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* RuntimeVersion with Latest and Version(u32)

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update polkadot.rs

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Adjust fetch_metadata to inspect version list

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* testing: Adjust metadata to metadata_legacy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* events: Adjust docs to use metadata_legacy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* have a pass over fetch_metadata

* cargo fmt

* Option<String> when fetch metadata via latest API

* clippy

* fmt

* cli: Use the MetadataVersion from codegen

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* cli: Specify latest as default for MetadataVersion

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* cli: Remove version from metadata and use the one from file_or_url

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Fix clippy

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* codegen: Decode metadata independently for different RPC calls

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: James Wilson <james@jsdw.me>
  • Loading branch information
@lexnv @jsdw
lexnv and jsdw authored May 3, 2023
1 parent f4eb80e commit 432e856
Show file tree
Hide file tree
Showing 33 changed files with 21,694 additions and 16,125 deletions.
2 changes: 2 additions & 0 deletions Cargo.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified artifacts/polkadot_metadata.scale
View file
Binary file not shown.
39 changes: 31 additions & 8 deletions cli/src/commands/compatibility.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ use frame_metadata::{
use jsonrpsee::client_transport::ws::Uri;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use subxt_codegen::utils::MetadataVersion;
use subxt_metadata::{get_metadata_hash, get_pallet_hash, metadata_v14_to_latest};

/// Verify metadata compatibility between substrate nodes.
Expand All @@ -25,16 +26,35 @@ pub struct Opts {
/// The validation will omit the full metadata check and focus instead on the pallet.
#[clap(long, value_parser)]
pallet: Option<String>,
/// Specify the metadata version.
///
/// - unstable:
///
/// Use the latest unstable metadata of the node.
///
/// - number
///
/// Use this specific metadata version.
///
/// Defaults to latest.
#[clap(long = "version", default_value = "latest")]
version: MetadataVersion,
}

pub async fn run(opts: Opts) -> color_eyre::Result<()> {
match opts.pallet {
Some(pallet) => handle_pallet_metadata(opts.nodes.as_slice(), pallet.as_str()).await,
None => handle_full_metadata(opts.nodes.as_slice()).await,
Some(pallet) => {
handle_pallet_metadata(opts.nodes.as_slice(), pallet.as_str(), opts.version).await
}
None => handle_full_metadata(opts.nodes.as_slice(), opts.version).await,
}
}

async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result<()> {
async fn handle_pallet_metadata(
nodes: &[Uri],
name: &str,
version: MetadataVersion,
) -> color_eyre::Result<()> {
#[derive(Serialize, Deserialize, Default)]
#[serde(rename_all = "camelCase")]
struct CompatibilityPallet {
Expand All @@ -44,7 +64,7 @@ async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result

let mut compatibility: CompatibilityPallet = Default::default();
for node in nodes.iter() {
let metadata = fetch_runtime_metadata(node).await?;
let metadata = fetch_runtime_metadata(node, version).await?;

match metadata.pallets.iter().find(|pallet| pallet.name == name) {
Some(pallet_metadata) => {
Expand Down Expand Up @@ -73,10 +93,10 @@ async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result
Ok(())
}

async fn handle_full_metadata(nodes: &[Uri]) -> color_eyre::Result<()> {
async fn handle_full_metadata(nodes: &[Uri], version: MetadataVersion) -> color_eyre::Result<()> {
let mut compatibility_map: HashMap<String, Vec<String>> = HashMap::new();
for node in nodes.iter() {
let metadata = fetch_runtime_metadata(node).await?;
let metadata = fetch_runtime_metadata(node, version).await?;
let hash = get_metadata_hash(&metadata);
let hex_hash = hex::encode(hash);
println!("Node {node:?} has metadata hash {hex_hash:?}",);
Expand All @@ -96,8 +116,11 @@ async fn handle_full_metadata(nodes: &[Uri]) -> color_eyre::Result<()> {
Ok(())
}

async fn fetch_runtime_metadata(url: &Uri) -> color_eyre::Result<RuntimeMetadataV15> {
let bytes = subxt_codegen::utils::fetch_metadata_bytes(url).await?;
async fn fetch_runtime_metadata(
url: &Uri,
version: MetadataVersion,
) -> color_eyre::Result<RuntimeMetadataV15> {
let bytes = subxt_codegen::utils::fetch_metadata_bytes(url, version).await?;

let metadata = <RuntimeMetadataPrefixed as Decode>::decode(&mut &bytes[..])?;
if metadata.0 != META_RESERVED {
Expand Down
42 changes: 35 additions & 7 deletions cli/src/utils.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
use clap::Args;
use color_eyre::eyre;
use std::{fs, io::Read, path::PathBuf};
use subxt_codegen::utils::Uri;
use subxt_codegen::utils::{MetadataVersion, Uri};

/// The source of the metadata.
#[derive(Debug, Args)]
Expand All @@ -16,29 +16,57 @@ pub struct FileOrUrl {
/// The path to the encoded metadata file.
#[clap(long, value_parser)]
file: Option<PathBuf>,
/// Specify the metadata version.
///
/// - unstable:
///
/// Use the latest unstable metadata of the node.
///
/// - number
///
/// Use this specific metadata version.
///
/// Defaults to 14.
#[clap(long)]
version: Option<MetadataVersion>,
}

impl FileOrUrl {
/// Fetch the metadata bytes.
pub async fn fetch(&self) -> color_eyre::Result<Vec<u8>> {
match (&self.file, &self.url) {
match (&self.file, &self.url, self.version) {
// Can't provide both --file and --url
(Some(_), Some(_)) => {
(Some(_), Some(_), _) => {
eyre::bail!("specify one of `--url` or `--file` but not both")
}
// Load from --file path
(Some(path), None) => {
(Some(path), None, None) => {
let mut file = fs::File::open(path)?;
let mut bytes = Vec::new();
file.read_to_end(&mut bytes)?;
Ok(bytes)
}
// Cannot load the metadata from the file and specify a version to fetch.
(Some(_), None, Some(_)) => {
// Note: we could provide the ability to convert between metadata versions
// but that would be involved because we'd need to convert
// from each metadata to the latest one and from the
// latest one to each metadata version. For now, disable the conversion.
eyre::bail!("`--file` is incompatible with `--version`")
}
// Fetch from --url
(None, Some(uri)) => Ok(subxt_codegen::utils::fetch_metadata_bytes(uri).await?),
(None, Some(uri), version) => Ok(subxt_codegen::utils::fetch_metadata_bytes(
uri,
version.unwrap_or_default(),
)
.await?),
// Default if neither is provided; fetch from local url
(None, None) => {
(None, None, version) => {
let uri = Uri::from_static("http://localhost:9933");
Ok(subxt_codegen::utils::fetch_metadata_bytes(&uri).await?)
Ok(
subxt_codegen::utils::fetch_metadata_bytes(&uri, version.unwrap_or_default())
.await?,
)
}
}
}
Expand Down
10 changes: 7 additions & 3 deletions codegen/src/api/calls.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -90,11 +90,11 @@ pub fn generate_calls(
pub fn #fn_name(
&self,
#( #call_fn_args, )*
) -> #crate_path::tx::Payload<#struct_name> {
) -> #crate_path::tx::Payload<types::#struct_name> {
#crate_path::tx::Payload::new_static(
#pallet_name,
#call_name,
#struct_name { #( #call_args, )* },
types::#struct_name { #( #call_args, )* },
[#(#call_hash,)*]
)
}
Expand All @@ -120,7 +120,11 @@ pub fn generate_calls(

type DispatchError = #types_mod_ident::sp_runtime::DispatchError;

#( #call_structs )*
pub mod types {
use super::#types_mod_ident;

#( #call_structs )*
}

pub struct TransactionApi;

Expand Down
23 changes: 21 additions & 2 deletions codegen/src/api/mod.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ mod calls;
mod constants;
mod errors;
mod events;
mod runtime_apis;
mod storage;

use frame_metadata::v15::RuntimeMetadataV15;
Expand All @@ -18,7 +19,7 @@ use crate::error::CodegenError;
use crate::{
ir,
types::{CompositeDef, CompositeDefFields, TypeGenerator, TypeSubstitutes},
utils::{fetch_metadata_bytes_blocking, Uri},
utils::{fetch_metadata_bytes_blocking, MetadataVersion, Uri},
CratePath,
};
use codec::Decode;
Expand Down Expand Up @@ -95,7 +96,11 @@ pub fn generate_runtime_api_from_url(
should_gen_docs: bool,
runtime_types_only: bool,
) -> Result<TokenStream2, CodegenError> {
let bytes = fetch_metadata_bytes_blocking(url)?;
// Fetch latest unstable version, if that fails fall back to the latest stable.
let bytes = match fetch_metadata_bytes_blocking(url, MetadataVersion::Unstable) {
Ok(bytes) => bytes,
Err(_) => fetch_metadata_bytes_blocking(url, MetadataVersion::Latest)?,
};

generate_runtime_api_from_bytes(
item_mod,
Expand Down Expand Up @@ -434,6 +439,14 @@ impl RuntimeGenerator {

let rust_items = item_mod_ir.rust_items();

let apis_mod = runtime_apis::generate_runtime_apis(
&self.metadata,
&type_gen,
types_mod_ident,
&crate_path,
should_gen_docs,
)?;

Ok(quote! {
#( #item_mod_attrs )*
#[allow(dead_code, unused_imports, non_camel_case_types)]
Expand Down Expand Up @@ -487,6 +500,12 @@ impl RuntimeGenerator {
TransactionApi
}

pub fn apis() -> runtime_apis::RuntimeApi {
runtime_apis::RuntimeApi
}

#apis_mod

pub struct ConstantsApi;
impl ConstantsApi {
#(
Expand Down
Loading

0 comments on commit 432e856

Please sign in to comment.