Gensis import duration is too much

[tac0turtle]

Today we upgraded the cosmos hub. Congrats to all the parties involved. In the process of importing the genesis file everyone observed wait times of upwards of 1 hour without invariant checks and some reported 5 hours with invariant checks. We should figure out why this is happening and try to speed this up. I don't see this as being a breaking change, so we could possibly sneak it into a Stargate minor release.

Opening this discussion board to discuss approaches to solving the issue. Once we have an agreed upon approach we can open the necessary issues.

[aaronc]

Fortunately we are now adding live migrations as of 0.42 so reimporting will hopefully be a thing of the past.

[tac0turtle]

There will need to be at lease 1-2 more of these until Tendermint implements non forking upgrades.

[aaronc]

Can you explain the issue with non forking upgrades on the Tendermint side @marbar3778 ?

[tac0turtle]

If we break the block protocol or p2p protocol chains will need to hard fork because there is no migration path even with the upgrade module. In the future we would like the chain to update the binary and the app say which versions of protocols to use.

[aaronc]

Is this because the on disk formats are breaking? How much does protobuf mitigate this?