handle forked blocks

[koushiro]

polkadot 0.8.28-1-99043f78-x86_64-linux-gnu:

2021-02-23 14:06:48  ✨ Imported #3909354 (0xd780…2e37)    
2021-02-23 14:06:53  💤 Idle (36 peers), best: #3909354 (0xd780…2e37), finalized #3909351 (0xd1d9…baae), ⬇ 56.0kiB/s ⬆ 77.9kiB/s    
2021-02-23 14:06:54  ✨ Imported #3909355 (0xd280…384d)    
2021-02-23 14:06:54  ♻️  Reorg on #3909355,0xd280…384d to #3909355,0x29be…d83a, common ancestor #3909354,0xd780…2e37    
2021-02-23 14:06:54  ✨ Imported #3909355 (0x29be…d83a)    
2021-02-23 14:06:58  💤 Idle (36 peers), best: #3909355 (0x29be…d83a), finalized #3909353 (0xbb73…5a5b), ⬇ 157.6kiB/s ⬆ 70.1kiB/s    
2021-02-23 14:07:00  ✨ Imported #3909356 (0x16a5…a1e6)    
2021-02-23 14:07:00  ♻️  Reorg on #3909356,0x16a5…a1e6 to #3909356,0xeb5d…d4c3, common ancestor #3909355,0x29be…d83a    
2021-02-23 14:07:00  ✨ Imported #3909356 (0xeb5d…d4c3)    
2021-02-23 14:07:03  💤 Idle (36 peers), best: #3909356 (0xeb5d…d4c3), finalized #3909353 (0xbb73…5a5b), ⬇ 102.9kiB/s ⬆ 158.8kiB/s    
2021-02-23 14:07:06  ✨ Imported #3909357 (0x6be2…000d)    
2021-02-23 14:07:06  ✨ Imported #3909357 (0xf838…394a)    
2021-02-23 14:07:08  💤 Idle (36 peers), best: #3909357 (0x6be2…000d), finalized #3909354 (0xd780…2e37), ⬇ 130.6kiB/s ⬆ 364.9kiB/s    

actual block 3909354 hash: 0xd780f986d5dfbad0236876940b23bef8b0abbba03ba503cf7a8288732d6d2e37
actual block 3909355 hash: 0x29be35af4c97f731c9607f78467b9cbc6213fa22ef2403c104976c1a6783d83a
actual block 3909356 hash: 0xeb5dd585e06635916228943f6d1eeae3074ed69efee0f97ddf9f314ebdbad4c3
actual block 3909357 hash: 0x6be2819b88e29250f1b6fedf27a6c809ac2f31ed104d78c165bf12deefe9000d

select * from blocks where block_num>=3909355 and block_num<=3909357;

   id    |                            parent_hash                             |                                hash                                | block_num |                             state_root                             |                          extrinsics_root                           |                                                                                                                                                                                           digest                                                                                                                                                                                           |                                                                                                                                                                 ext                                                                                                                                                                  | spec 
---------+--------------------------------------------------------------------+--------------------------------------------------------------------+-----------+--------------------------------------------------------------------+--------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------
 3909356 | \xd780f986d5dfbad0236876940b23bef8b0abbba03ba503cf7a8288732d6d2e37 | \x29be35af4c97f731c9607f78467b9cbc6213fa22ef2403c104976c1a6783d83a |   3909355 | \x74fc760ae5e9d7346db6f3ef244d27cdb7c59edaee423ef82946967362e5a54f | \xda51021301c28c817b81691c2d3cb5d447009d12bb0adecfad1a71e5c1e0a96f | \x080642414245b501017600000095c408100000000080025fdc6666c3bcc56024d6c1db2092d729c20d50d5b88be36a9bef9cedc21ab54a737c4a9c2622b1cc12d0cf867967490e65d24dc2e2d785e35f55af9fea0c9f3d85b1906e78f62ae4f5229697d35fbb7dde555a7aba0112bed250de198d0a054241424501012e746a2b33c51594a24bbc752b8c5232402f7a909ca6a182bd08cbb61a42f44b5588688ff531bcd590b8dda8c322e6dd02f8aa6e2c03c4ac31afa0297361be8b | \x08280403000b30647fcd77014d02840056daf75bd0f09c11d798263bc79baeb77c4b4af1dbd372bbe532b1f8702b2a7e01463c543bb8d0a16c0914d18073ccb0ab0359341ff2e784079a552bfb7f06290b0679fd4788f36bb6a940907e888d8999a41f109ce254ad107ba2f4fd0b4b6e867502d2770200000500004f89f980a0c7bf775a70d8966672c89c564c16c9564af03c5338b2a35af1aacb0700e40b5402 |   28
 3909357 | \x29be35af4c97f731c9607f78467b9cbc6213fa22ef2403c104976c1a6783d83a | \x16a511d2aae56d02fac76b3553a0e45d5c084646dd60b54d5aa3e0d56f28a1e6 |   3909356 | \x27392821ff1bb88d5f019811ef4ff9574e6ac510f742df236fddeaf0e5e167b3 | \x6fb822c6d9663c7e8f72d1d8aa8e6b7e886730a50eba107a6b64dc1ad46467a9 | \x080642414245b501033400000096c40810000000001edb813339135b0ddf3276782f6182fef18e2cb2c3d9dc2042f8514a8ced0e023bb9eed874ca1d7b9248654b22dc1a1ece5928beabc6b08c411a2c79ef17920807a774c63b3ed26fde48dd363c2b0e2fe5121d6e86f1657e21373e1613bf940e0542414245010150314bff108eae652deef80915846bb13f447dbec89dcd4b71960bca5259661b75a87b9813970599789c309e9530dee65232ff2f4f63254e31385b9bf85f2881 | \x04280403000ba07b7fcd7701                                                                                                                                                                                                                                                                                                           |   28
 3909358 | \xeb5dd585e06635916228943f6d1eeae3074ed69efee0f97ddf9f314ebdbad4c3 | \x6be2819b88e29250f1b6fedf27a6c809ac2f31ed104d78c165bf12deefe9000d |   3909357 | \x2a54e5f5d0451322348462b679174549f1a95b8f5619f315d03054933b946397 | \x328b9c0b56c94066be01a4938b42dcd502877376c6045db591513ea44806c8a2 | \x080642414245b50101be00000097c40810000000000045bc5be27826f1fe633145b259d3495d102cd3552cdf5a726e8dbffbb32e60d69e8a6e044b08cdf35ed87ad0cfbe8f2737c19f45f91e6dd365a9e61d374405d27410bece5e54eaf74d5f681f8d34d86343ee9eec01d031dd2414659cd22803054241424501015eef40c5f422a6063cc4af2755a3b43e106f9636e56e641422e6364f12c2f42ca29c3cad529d584671689af1c8e0bf07cc7d1e4772b0db35acd59b6cf0510983 | \x04280403000b10937fcd7701                                                                                                                                                                                                                                                                                                           |   28

[koushiro]

any ideas about it? @insipx

[insipx]

currently trying to reproduce currently on my archive node. Might require some further investigation into substrate to see if anything has changed with regards to block finalization and storage in rocksdb. It's my hope that we can let substrate handle this, rather than dealing with reorgs within substrate-archive. But needs more research I think. A quick glance at this also lead me to believe it might just be an off-by-one somewhere along the path of blocks being inserted into postgres. Might be best to rule that out first before looking into reorgs, however since the pattern does not continue for anything but 3909355-3909356 3909357-3909358, leads me to believe that it is reorgs.

I also got different values for blocks 3909354 from subscan than you listed:

Subscan Values:
3909354: 0xd780f986d5dfbad0236876940b23bef8b0abbba03ba503cf7a8288732d6d2e37
3909355: 0xd28060f41e2b7854a04a4c75c922ad096bda8f875838095360995a96054c384d
3909356: 0x16a511d2aae56d02fac76b3553a0e45d5c084646dd60b54d5aa3e0d56f28a1e6
3909357: 0x6be2819b88e29250f1b6fedf27a6c809ac2f31ed104d78c165bf12deefe9000d

Interestingly again, Polkascan can't seem to agree on the values either:

Polkascan
3909354: 0xd780f986d5dfbad0236876940b23bef8b0abbba03ba503cf7a8288732d6d2e37
3909355: 0x29be35af4c97f731c9607f78467b9cbc6213fa22ef2403c104976c1a6783d83a
3909356: 0xeb5dd585e06635916228943f6d1eeae3074ed69efee0f97ddf9f314ebdbad4c3
3909357: 0x6be2819b88e29250f1b6fedf27a6c809ac2f31ed104d78c165bf12deefe9000d

Querying the RPC on my polkadot archive node, I got the same values you listed so those are definitely the correct ones

[insipx]

The easiest way to ensure we only index finalized blocks is not indexing anything beyond the latest finalized block:

substrate-archive/substrate-archive-backend/src/read_only_backend/blockchain_backend.rs

Line 68 in 30ecb33

fn last_finalized(&self) -> ChainResult<Block::Hash> {

We could just let reorgs take place, and only keep up with the last finalized block. I think this issue only arises once Archive is caught up to the running chain already, so it shouldn't matter too much if we're a couple blocks behind, and avoids all the logic or updating/reinserting blocks that are being reorged