Disk-backed externalities

[liamaharon]

Original issue paritytech/substrate#14076

---

Prelude

Discussion in paritytech/substrate#13562 expanded to include various topics other than lazy-download, including improving general performance and using a disk-backed database.

I am creating this issue to separate the discussion and progress around allowing the try-runtime-cli to accept a DB path for reading state, rather than loading all state into memory.

Motivation

Creating snapshots and loading state into memory is not a significant issue for chains like Polkadot and Kusama, but it can be for heavier-weight chains like Moonbeam with extensive state.

On these heavy chains:

1. It takes a long time to create a snapshot, and the snapshot size is considerable.
2. The state might not fit into memory, depending on the chain and machine specifications.
3. It would provide a much better developer experience for using try-runtime-cli with these chains if developers could skip worrying about that and just specify the path to their existing database instead.

Implementation ideas

• TestExternalities currently has a hardcoded InMemoryBackend as a generic parameter. However, that could be updated to a more generic sp_state_machine::Backend.
• Once that change is made, we should be able to use either an InMemoryBackend or DiskDbBackend (which is already implemented in fudge) in TestExternalities.
• We can add a new Mode to remote-externalities called something like Disk, so we would have Online, Offline (perhaps rename this to Snap), and Disk.
• Finally, we can update the CLI to accept live, snap, and a new option db when specifying where to get state. If the user chooses db, they pass the path, and the TestExternalities will be initialized with a DiskDbBackend instead of the InMemoryBackend.