Check this Rust Library for Similar Purpose

[Congyuwang]

Rust Bitcoin Deserialization Library

https://github.com/Congyuwang/Rusty-Bitcoin-Explorer

Performance

Deserializing the whole chain takes 10 minutes (SSD, Core i7 4-core).
Deserializing the whole chain with outpoint connected to outputs takes 30 minutes (SSD, Core i7 4-core).

[vincenzopalazzo]

Hi @Congyuwang,

Cool, thanks to share this project, with rust-bitcoin are born a lot of parsers, and with the primitive it is easy.

Deserializing the whole chain takes 10 minutes (SSD, Core i7 4-core).
Deserializing the whole chain with outpoint connected to outputs takes 30 minutes (SSD, Core i7 4-core).

Here you are missing important details if you want to compare your parser with my bachelor thesis that it is the amount of RAM used in that parser phase. SpyCBlock is only a starting point to prove that you can decode the blockchain with less than 1 Gb of ram, this means that you can try to decode a couple of blocks also on a raspberry pi model 2 that is history now.

if you have a remote machine where you make benchmarking of your project, I have a clean branch where I was working last year, but I never finish that I would like to test, in terms of speed and space of the result.

Thanks again to share this project.

[Congyuwang]

Hi @vincenzopalazzo,

Thank you for your reply!

Cool, decoding with less than 1GB RAM is quite impressive. I tried to reduce the amount of memory used. To just decode blocks per se seems to require not much RAM. With my current library, my crate will need about 2-3GB RAM, mainly to keep a copy of block index (map from block hash to block file number and offset) for each thread. I could reduce this a bit by giving each thread a read-only reference to this index.

However, I find it difficult to reduce RAM when I need to find out the addresses of the inputs of transactions because I need to keep track of unspent transactions in memory. This in-memory UTXO cache takes 30GB RAM.

[vincenzopalazzo]

To just decode blocks per se seems to require not much RAM. With my current library, my crate will need about 2-3GB RAM, mainly to keep a copy of the block index (map from block hash to block file number and offset) for each thread. I could reduce this a bit by giving each thread a read-only reference to this index.

The disk is the bottleneck of SpyCblock, I never test it on the SSD, because time is not important at all for me, when you parse a big amount of data, you want useful information to make your analysis on. Spycblock generates a JSON version of the blockchain where you could make your analysis on. I think you discovered already this project :)

But with SpyCBlock I try to answer, with X amount of time how much useful data I can generate?.

I find it difficult to reduce RAM when I need to find out the addresses of the inputs of transactions because I need to keep track of unspent transactions in memory.

You need to just parse the leveldb content :) it is much smarter than parser all blockchain. but here rust-bitcoin can't help I think.

This in-memory UTXO cache takes 30GB RAM.

IMO, the best choice is already taken from Bitcoin core, that is store the data on the disk. it takes more time, but it is the correct thing to do.

[Congyuwang]

Yes. I guess doing a one-time conversion and storing useful information in JSON before further analysis is a good idea.

Computing everything on the go is very resource intensive.

[vincenzopalazzo]

I agree, @Congyuwang.

If you think about the time that bitcoin core takes double the space that it use on the disk it is doubled from 2020 to now.

[Congyuwang]

Great. I updated my library to use RocksDB (instead of Hash-map) to keep track of UTXO. Now it needs at most 3-4 GB RAM and finishes an iteration through all blocks in about 2.5 hours.

[vincenzopalazzo]

@Congyuwang, as you can see the performance is growing, and you are storing a hash 256 and a counter (uin64?). If you store all the structure or some other fancy serialization your memory will increase because the first it is the bottolneck.

But any way. It is a good discussion, thanks to share these details, and don't stop study Bitcoin.

[Congyuwang]

@vincenzopalazzo I am storing a 128 bit rehashing of the original 256 hash to save space. For now there are fewer than 700 million transactions, and the probability of having at least one hash collision using 128 bit hash is very very small (p < 10e-18, according to https://en.m.wikipedia.org/wiki/Birthday_problem). I am also only keeping a u32 counter, so added together it gives me a (128 + 32) 20-byte key for indexing txouts.

[vincenzopalazzo]

I disagree with the usage of probability here, because fixing a probability error when happens has a cost.

I will preferer a persistent data structure, but it is only an opinion.

(p < 10e-18, according to https://en.m.wikipedia.org/wiki/Birthday_problem).

Can you prove that with some benchmarking like the following one on the bloom filter? https://github.com/vincenzopalazzo/cpstl/blob/master/cpp/data_structures/bloomfilter/benchmark/Main.cpp