libmetachain FFI integration

Continuing from libain gRPC integration, we move to integrating metachain through FFI for providing a safe passage to communicate across defichain-metachain boundary.

Background

Metachain (DMC) almost entirely relies on defichain for consensus.

Overall flow:

• Node packages transactions and mines a block
• DMC MintBlock is called with the relevant transactions (empty if none are related to DMC)
• DMC returns a serialized block as payload (this method is not fallible, except maybe temporarily)
• Encoded payload gets embedded into CBlock
• ConnectBlock validation happens in native chain, DMC ConnectBlock gets called, block gets connected and persisted in storage.

There are two ways to invoke DMC - one is through RPC and the other involves FFI. While RPC makes it possible to keep meta and native chain separate, it's not suitable for production as it'd mean that users have to run both the executables and RPC is prone to failure as it relies on host network. RPC is also less secure because we have to expose control RPCs at host network. Going the FFI route makes things easier and secure - it packages DMC inside defid and functions are invoked from within memory which makes it more reliable. That said, we still need RPC for e2e testing, so we need libain for generating RPC client code and we should also compile libmetachain as a static library which can then be linked to defid through depends system (two linked libraries).

At runtime, defichain instantiates metachain (using -meta <args> or -meta_rpc <url>) and once its network boots up, defid goes about its normal workflow. If either of the daemons fail, the program will exit.

Protobuf

We start with defining the relevant RPCs in libain-rs protobuf, at least those which need to be invoked from defichain. We use protobuf to adhere RPCs and FFI functions to follow a spec. Taking MintBlock as an example:

syntax = "proto3";
package rpc;

// Message field names/types should match the struct field names/types in metachain

message MetaTransaction {
    string from = 1;
    string to = 2;
    int64 amount = 3;
}

message MetaBlockInput {
    repeated MetaTransaction txs = 2;
}

message MetaBlockResult {
    bytes payload = 1;
}

service Metachain {
    // [rpc: metaConsensusRpc_mintBlock] [client] <-- specifying metachain URL for mint block and generating only client code
    rpc MetaMintBlock(types.MetaBlockInput) returns (types.MetaBlockResult);
}

This would generate the necessary JSON RPC FFI functions which can then be called from defid (through FFI), which will send an RPC request through the async runtime in Rust.

Similarly, we define structures and functions in metachain which are exposed through FFI using #[cxx::bridge] macro:

#[cxx::bridge]
mod ffi {
    pub struct DmcTx {
        pub from: String,
        pub to: String,
        pub amount: i64,
    }

    pub struct DmcBlock {
        pub payload: Vec<u8>,
    }

    extern "Rust" {
        fn mint_block(dmc_txs: &CxxVector<DmcTx>) -> Result<DmcBlock>;
    }
}

We also emit the necessary C++ header/source files at build time, which must be included in order for us to call the above functions. The header/source files are automatically included using ./make.sh patch-codegen

if (is_ffi) {
    try {  // #include<libmc.h>
        auto block = mint_block(txs);
    } catch (const std::exception& e) {
        // error
    }
} else {
    try {  // #include<libain_rpc.h>
        auto client = NewClient(rpc_url);
        auto block = CallMetaMintBlock(client, inp);
    } catch (const std::exception& e) {
        // error
    }
}

Static lifetimes

In order to access the RPC URL/client inside defid and WASM client inside DMC, we rely on static variables (lazy_static! in Rust). They get initialized as soon as the executable starts (before starting either network), so we can expect them to hold a value anywhere in the middle of the control flow.

Moving funds across bridge

In order to send funds to metachain, a custom tx is created (DepositToMetachain) where the to address is ignored by defichain and the funds are instead sent to a locked address. DMC reacts to this and mints DFI tokens. Similarly, a WithdrawFromMetachain custom tx is used for getting funds back from lock address after DMC burns the tokens.