wasmi: Substrate Integration

[Robbepop]

There has been a lot of work ongoing for the wasmi WebAssembly interpreter since quite a while. Multiple new releases have been publishes that significantly improve the API as well as the compilation and execution performance. Some Wasm proposals such as multi-value have been implemented, the codebase has been modernized and a few bugs have been fixed.

However, due to the changes in the wasmi API that made some of the improvements possible porting the new wasmi versions to Substrate unfortunately requires some work on the Substrate side, too.

The Vision

We want to integrate all of these improvements into Substrate and are primarily working towards this goal.
This will significantly improve the smart contract story on Substrate, ink! and any other related project.

Parity has long been working on improving the performance of the sandboxed Wasm smart contracts execution engine.
In the past some experiments have already been performed using the Wasmer JIT engine, however, the conclusion was that it does not actually fit our use case where smart contracts are usually pretty small Wasm blobs with only a limited amount of computation done per invocation. We might explore this space more and see whether these initial conclusions are correct but until then our mission is to optimize wasmi for our smart contract uses cases as much as we possibly can.

The implementation of new WebAssembly proposals allow us to be more flexible with the generation of the smart contract Wasm blobs and might even reduce file sizes slightly.

The Plan

• Implement the new wasmi with its many new optimizations, new supported Wasm proposals, new API, codebase modernizations and bug fixes.
  • This item encompasses way too many past PRs on the wasmi repository than I am willing to link here.
  • Some outstanding (optional) optimizations include these:
    • Optimization: Inline caching of the top-most stack value wasmi-labs/wasmi#417
    • Optimization: Bytecode dispatching via function pointers wasmi-labs/wasmi#424
    • Make use of wasmparser::VisitOperator API wasmi-labs/wasmi#418
• Release a version of the new wasmi implementation on crates.io that is capable of supporting our demands for Wasm smart contract executions.
  • Link: https://crates.io/crates/wasmi
  • The newer version of wasmi is based on the wasmparser framework instead of providing its own Wasm parsing and validation infrastructure. While this benefits us greatly in the long run it requires us to maintain our of fork of the wasmparser repository in order to support embedded no_std environments. This is required since we primarily run wasmi inside a WebAssembly sandbox without standard library support. Furthermore since wasmparser itself depends on indexmap we also have to maintain a similar no_std fork for it, too.
    • wasmparser-nostd: https://crates.io/crates/wasmparser-nostd
    • indexmap-nostd: https://crates.io/crates/indexmap-nostd
• Test the new wasmi version in-depth. For this we want to fuzzy test wasmi execution and compare it to the official Wasm spec interpreter that is assumed to be working correctly. This shall provide us with the confidence we require to use the new wasmi implementation in production use cases.
  • Fortunately we have setup a decent collaboration between the wasmi team and the BytecodeAlliance team that is maintaining the Rust bindings for the official Wasm spec interpreter.
  • The downside is that the Wasm spec interpreter itself is implemented in OCaml and therefore requires some intricate bridging between Rust and the OCaml runtime.
  • The Wasmtime project already fuzz tests against an old version of wasmi. Ideally we could update the wasmi version used there to catch bugs on both sides for the newer wasmi implementation.
    • Link: https://github.com/bytecodealliance/wasmtime/blob/573ae0c60bb20e010d74ba7fd2e7e59247f3385d/crates/fuzzing/src/oracles/diff_wasmi.rs
  • Issue: Implement fuzz testing for Wasm execution wasmi-labs/wasmi#419
  • wasm-spec-interpreter Rust Bindings: https://github.com/bytecodealliance/wasm-spec-interpreter
• Support the new wasmi API for Substrate's sp-sandbox so that we can actually make use of the new wasmi implementation. The good thing is that the new wasmi API pretty much resembles the well-known API of the Wasmtime execution engine.
  • Issue: Refactor sp-sandbox to accomodate new wasmi API substrate#11752

Open Questions

None so far.

[athei]

I ticked the last point because we are not integrating into sp-sandbox. We removed sp-sandbox and go with an in-runtime wasmi. The new wasmi is already in use.

I think we can close here? @Robbepop

[Robbepop]

@athei Yes I think we can close this issue.