IAVL snapshot - pruning race issue tracking

[p0mvn]

This is an issue to track our tests related to testing various configurations for snapshot and pruning settings on v6.4.0.

Our test cases:
https://docs.google.com/spreadsheets/d/1hTnM8GPnSuD0wppgNNWbKss7buLL_RKS0jaYo0X9EkM/edit#gid=0

[gabriellahayden]

Are you still having issues tracking the pruning race?

[p0mvn]

We are testing various configurations outside of the recommended settings. There are currently no observed issues on the recommended settings.

With these tests, our goal is to fix the problem that requires to keep a large number of pruning-keep-recent

[p0mvn]

Currently, we don't know what is the threshold of prunining-keep-recent that causes pruning to catch up with snapshot and fail. This is what we are trying to figure out with these tests.

[gabriellahayden]

Okay

[p0mvn]

Context
Currently, we know that node runners tend to get an error here when rigorous pruning is enabled:
https://github.com/osmosis-labs/iavl/blob/2e0b26adfd2552db7dc380a078a83dc53738486f/nodedb.go#L509

Potential solution
This problem can be fixed by introducing a map in nodedb called versionsToPrune map[int64]struct{}. If a version X is currently under snapshot, instead of erroring out, we could attempt to queue that version in the map. When the snapshot is done, we check the map and then:

• if a version exists, we prune it
• if a version does not exist, we continue normally

More issues related to pruning and snapshot are possible.

Next Steps

To ensure, that we are confident that the identified solution can help:

• set up a node with rigorous pruning settings
• observe and document all errors
• evaluate if the Potential Solution still applies
• document next steps

[ValarDragon]

Another option is seeing if we can disable snapshots when syncing SDK-side

[p0mvn]

The problem occurs not only during synching when aggressive pruning and snapshots are enabled

Here we have:

pruning-keep-recent = "100"
pruning-keep-every = "0"
pruning-interval = "10"

snapshot-interval = 100
snapshot-keep-recent = 2

E.g.:

�[90m11:29PM�[0m �[32mINF�[0m commit start, height - 3416920
�[90m11:29PM�[0m �[32mINF�[0m commit end, height - 3416920
�[90m11:29PM�[0m �[32mINF�[0m prune start, height - 3416920

�[90m11:29PM�[0m �[32mINF�[0m prune end, height - 3416920

�[90m11:29PM�[0m �[32mINF�[0m flushing metadata
�[90m11:29PM�[0m �[32mINF�[0m flushing metadata finished
�[90m11:29PM�[0m �[1m�[31mERR�[0m�[0m CONSENSUS FAILURE!!! �[36merr=�[0m"unable to delete version 3416900 with 1 active readers" �[36mmodule=�[0mconsensus �[36mstack=�[0m"goroutine 920 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:726 +0x4c\npanic({0x1be1e80, 0xc17a6fc210})\n\truntime/panic.go:1038 +0x215\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit.func1()\n\tgithub.com/cosmos/cosmos-sdk@v0.45.1/store/rootmulti/store.go:398 +0xcb\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0001ba600)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.1/store/rootmulti/store.go:423 +0x46e\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc0010acea0)\n\tgithub.com/cosmos/cosmos-sdk@v0.45.1/baseapp/abci.go:308 +0x231\ngithub.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc000d58ae0)\n\tgithub.com/tendermint/tendermint@v0.34.15/abci/client/local_client.go:264 +0xb6\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0x0)\n\tgithub.com/tendermint/tendermint@v0.34.15/proxy/app_conn.go:93 +0x22\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc000da43b0, 0x7}}, {0xc000da43c0, 0xb}, 0x310f7e, 0x342358, {{0xc05f977460, ...}, ...}, ...}, ...)\n\tgithub.com/tendermint/tendermint@v0.34.15/state/execution.go:228 +0x269\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc000da43b0, 0x7}}, {0xc000da43c0, 0xb}, 0x310f7e, 0x342358, {{0xc05f977460, ...}, ...}, ...}, ...)\n\tgithub.com/tendermint/tendermint@v0.34.15/state/execution.go:180 +0x6ee\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc00012ca80, 0x342358)\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:1635 +0x9fd\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc00012ca80, 0x342358)\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:1546 +0x305\ngithub.com/tendermint/tendermint/consensus.(*State).addProposalBlockPart(0xc00012ca80, 0x3d1, {0xc0023f8300, 0xc00237c400})\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:1919 +0x92e\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc00012ca80, {{0x2774200, 0xc11e4d77e8}, {0xc0023f8300, 0x0}})\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:820 +0x17c\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc00012ca80, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:762 +0x419\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:378 +0x12f\n"
�[90m11:29PM�[0m �[32mINF�[0m Stopping baseWAL service �[36mimpl=�[0m{"Logger":{}} �[36mmodule=�[0mconsensus �[36mwal=�[0m/root/.osmosisd/data/cs.wal/wal
�[90m11:29PM�[0m �[32mINF�[0m Stopping Group service �[36mimpl=�[0m{"Dir":"/root/.osmosisd/data/cs.wal","Head":{"ID":"4YyJFPup8UKp:/root/.osmosisd/data/cs.wal/wal","Path":"/root/.osmosisd/data/cs.wal/wal"},"ID":"group:4YyJFPup8UKp:/root/.osmosisd/data/cs.wal/wal","Logger":{}} �[36mmodule=�[0mconsensus �[36mwal=�[0m/root/.osmosisd/data/cs.wal/wal
�[90m11:29PM�[0m �[32mINF�[0m Timed out �[36mdur=�[0m3000 �[36mheight=�[0m3416920 �[36mmodule=�[0mconsensus �[36mround=�[0m0 �[36mstep=�[0m3
�[90m11:29PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:29PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mp2p
�[90m11:29PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mp2p
�[90m11:29PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mp2p
�[90m11:29PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 5): incompatible: peer is on a different network. Got osmo-testnet-1, expected osmo-test-4 �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 5): dial tcp 165.227.122.46:26656: i/o timeout �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:29PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 4): dial tcp 157.90.81.24:26656: i/o timeout �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 159.69.193.212:26656: connect: connection refused �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 142.132.151.115:26656: connect: connection refused �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): incompatible: peer is on a different network. Got osmo-test-3, expected osmo-test-4 �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 138.197.2.186:26656: i/o timeout �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 54.74.184.239:26656: i/o timeout �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 54.74.184.239:26656: i/o timeout �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 142.93.177.164:26656: i/o timeout �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 159.203.100.232:26656: i/o timeout �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 65.108.79.246:26680: i/o timeout �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 5): dial tcp 157.90.81.24:26656: i/o timeout �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 68.183.153.16:26656: i/o timeout �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mp2p
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): incompatible: peer is on a different network. Got osmo-testnet-1, expected osmo-test-4 �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:30PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 165.227.122.46:26656: i/o timeout �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Saving AddrBook to file �[36mbook=�[0m/root/.osmosisd/config/addrbook.json �[36mmodule=�[0mp2p �[36msize=�[0m16
�[90m11:31PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 6): dial tcp 157.90.81.24:26656: i/o timeout �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mp2p
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 159.69.193.212:26656: connect: connection refused �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 142.132.151.115:26656: connect: connection refused �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): incompatible: peer is on a different network. Got osmo-test-3, expected osmo-test-4 �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 65.108.79.246:26680: i/o timeout �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 138.197.2.186:26656: i/o timeout �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 159.203.100.232:26656: i/o timeout �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 54.74.184.239:26656: i/o timeout �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 68.183.153.16:26656: i/o timeout �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 142.93.177.164:26656: i/o timeout �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:31PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 54.74.184.239:26656: i/o timeout �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:32PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mp2p
�[90m11:32PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mp2p
�[90m11:32PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): incompatible: peer is on a different network. Got osmo-testnet-1, expected osmo-test-4 �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 165.227.122.46:26656: i/o timeout �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:32PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:32PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mp2p
�[90m11:32PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 7): dial tcp 157.90.81.24:26656: i/o timeout �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Saving AddrBook to file �[36mbook=�[0m/root/.osmosisd/config/addrbook.json �[36mmodule=�[0mp2p �[36msize=�[0m16
�[90m11:33PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:33PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8882f94f5fbbf17ce445d3190ae494bf12a93857","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:33PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:34PM�[0m �[32mINF�[0m Ensure peers �[36mmodule=�[0mpex �[36mnumDialing=�[0m0 �[36mnumInPeers=�[0m0 �[36mnumOutPeers=�[0m3 �[36mnumToDial=�[0m37
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"8e59cd989b8aa826d8d031b672eaa20f893a46fc","ip":"157.90.81.24","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ef9264db437aaf715cc5ca34e995ddf23ecb153c","ip":"164.90.155.120","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"e159391f00e8127d8e6ec1319b04633ffc33ed1a","ip":"165.227.122.46","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m Will dial address �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[32mINF�[0m We need more addresses. Sending pexRequest to random peer �[36mmodule=�[0mpex �[36mpeer=�[0m{"Data":{},"Logger":{}}
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[32mINF�[0m Dialing peer �[36maddress=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mp2p
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): dial tcp 142.132.151.115:26656: connect: connection refused �[36maddr=�[0m{"id":"2b5d9070f206c4ba1b495fe12486387495f04524","ip":"142.132.151.115","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): dial tcp 159.69.193.212:26656: connect: connection refused �[36maddr=�[0m{"id":"d8c88ca91a05c408f102ce7a1804204b933058e2","ip":"159.69.193.212","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): incompatible: peer is on a different network. Got osmo-test-3, expected osmo-test-4 �[36maddr=�[0m{"id":"298b05b723526957112c973eed2b36706d7cdee4","ip":"157.90.81.24","port":7000} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): dial tcp 68.183.153.16:26656: i/o timeout �[36maddr=�[0m{"id":"edd2b4968f012148641205b8ddd29f1beae8ab09","ip":"68.183.153.16","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): dial tcp 138.197.2.186:26656: i/o timeout �[36maddr=�[0m{"id":"ee436b6d1ecac2dd78498f420626e498abb7bd95","ip":"138.197.2.186","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): dial tcp 142.93.177.164:26656: i/o timeout �[36maddr=�[0m{"id":"b894030ba5cf2dab4fbe092eb659004d70d7dc90","ip":"142.93.177.164","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): dial tcp 65.108.79.246:26680: i/o timeout �[36maddr=�[0m{"id":"e2c6a63fc3b1a2722b4b7628d03f80d7ad6bc9dd","ip":"65.108.79.246","port":26680} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): dial tcp 54.74.184.239:26656: i/o timeout �[36maddr=�[0m{"id":"f7f66d06e179ef2b2abc5b2ea6e9280c7edb868a","ip":"54.74.184.239","port":26656} �[36mmodule=�[0mpex
�[90m11:34PM�[0m �[1m�[31mERR�[0m�[0m dialing failed (attempts: 8): dial tcp 159.203.100.232:26656: i/o timeout �[36maddr=�[0m{"id":"f5051996db0e0df69c55c36977407a9b8f94edb4","ip":"159.203.100.232","port":26656} �[36mmodule=�[0mpex
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdc54d3]

goroutine 1698 [running]:
github.com/cosmos/iavl.(*nodeDB).incrVersionReaders(0x27bf1e8, 0x20)
	github.com/cosmos/iavl@v0.17.3/nodedb.go:930 +0x33
github.com/cosmos/iavl.newExporter(0xc0ced4aea0)
	github.com/cosmos/iavl@v0.17.3/export.go:45 +0xc8
github.com/cosmos/iavl.(*ImmutableTree).Export(...)
	github.com/cosmos/iavl@v0.17.3/immutable_tree.go:142
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Export(0xc09b6da060, 0x2787580)
	github.com/cosmos/cosmos-sdk@v0.45.1/store/iavl/store.go:245 +0x52
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Snapshot.func2()
	github.com/cosmos/cosmos-sdk@v0.45.1/store/rootmulti/store.go:724 +0x4b8
created by github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Snapshot
	github.com/cosmos/cosmos-sdk@v0.45.1/store/rootmulti/store.go:691 +0x53b

[p0mvn]

What is the most aggressive pruning that we would like to support while having snapshots enabled?

[ValarDragon]

Oh interesting, I didn't realize snapshots are enabled here. I think we have to require keep-recent to be a sizable multiple of the snapshot interval right? Is there a way around that?

[p0mvn]

That's correct. I think there is a way where we can work around requiring this keep-recent buffer. Here is my current understanding of the potential approaches and trade-offs:

1. One idea is the one I mentioned here where we use a map to check whether the version is currently being pruned before taking the snapshot and v.v. However, synchronizing that would be tough. I will try creating a spike and see how it performs.

2. Another way, which is the simplest is using the pruning-keep-every flag in the config. However, then the disk size would always grow. You also mentioned that it is going to be removed in the upcoming SDK release. So that might not be it.

3. Previously, I also thought of blocking the execution flow in prune until the snapshot finishes. Made a spike here. However, I don't think anymore that this is a good approach since we would be blocking for long waiting for the snapshot to finish. It also doesn't account for the case if we try to take a snapshot of an already pruned version (though that can be added). In addition, the change was quite difficult to synchronize and is quite risky.

[p0mvn]

Just an update on this. I made a modified UML class diagram to understand the coupling between prune and snapshot logic better:

Need to spend a bit more time thinking about the final design and all edge cases.

Currently, I'm leaning towards the following:

• a snapshot.Manager pointer passed to types.CommitMultiStore and stored as a field
• snapshot() method moved from Baseapp to types.CommitMultiStore
• WIP: Some logic similar to 1) in the previous message implemented in types.CommitMultiStore

The main idea is to move snapshot logic closer to where we perform pruning. Currently, these 2 functionalities are highly coupled but are done at different abstraction layers. As soon as we have them together, it would be easier to fix the racing problem.

I will be back with a final design update this weekend.

[p0mvn]

Had a bit of a delay on this, ended up being more complex than I thought.

However, the implementation is ready here: osmosis-labs/cosmos-sdk#140

Everything I have so far is documented in the PR above. It seems to be working as I would expect it with pruning-everything and snapshot-interval=100. I'm leaving my node to run overnight to see if it crashes and by how much the disk usage changes.

I still need to do a minor cleanup of the code. Expecting to open a review tomorrow morning if my test node doesn't crash

[p0mvn]

This was merged