Weekly Digest (19 January, 2020 - 26 January, 2020)

[weekly-digest]

Here's the Weekly Digest for status-im/status-go:

---

ISSUES

Last week 8 issues were created.
Of these, 4 issues have been closed and 4 issues are still open.

OPEN ISSUES

💚 #1811 include mail password when getting enode with get_enode.sh, by jakubgs
💚 #1809 fix: close resultsets so we don't leak them, by andremedeiros
💚 #1807 Bump github.com/ethereum/go-ethereum from 1.9.5 to 1.9.10, by dependabot-preview[bot]
💚 #1806 Bump github.com/pkg/errors from 0.8.1 to 0.9.1 in /protocol, by dependabot-preview[bot]

CLOSED ISSUES

❤️ #1813 Fix wrong field being logged in EnvelopesMonitor, by pedropombeiro
❤️ #1812 bump to 0.40.0, by rasom
❤️ #1810 Fix ens names and contact, by cammellos
❤️ #1808 v0.39.8, by cammellos

NOISY ISSUE

🔈 #1812 bump to 0.40.0, by rasom
It received 4 comments.

---

PULL REQUESTS

Last week, 17 pull requests were created, updated or merged.

UPDATED PULL REQUEST

Last week, 8 pull requests were updated.
💛 #1811 include mail password when getting enode with get_enode.sh, by jakubgs
💛 #1809 fix: close resultsets so we don't leak them, by andremedeiros
💛 #1807 Bump github.com/ethereum/go-ethereum from 1.9.5 to 1.9.10, by dependabot-preview[bot]
💛 #1806 Bump github.com/pkg/errors from 0.8.1 to 0.9.1 in /protocol, by dependabot-preview[bot]
💛 #1796 Bump github.com/cenkalti/backoff/v3 from 3.0.0 to 3.2.2 in /protocol, by dependabot-preview[bot]
💛 #1756 Bump github.com/prometheus/client_golang from 1.2.1 to 1.3.0, by dependabot-preview[bot]
💛 #1737 Bump github.com/libp2p/go-libp2p-core from 0.2.3 to 0.3.0, by dependabot-preview[bot]
💛 #1735 Bump github.com/multiformats/go-multiaddr from 0.1.1 to 0.2.0, by dependabot-preview[bot]

MERGED PULL REQUEST

Last week, 9 pull requests were merged.
💜 #1813 Fix wrong field being logged in EnvelopesMonitor, by pedropombeiro
💜 #1812 bump to 0.40.0, by rasom
💜 #1810 Fix ens names and contact, by cammellos
💜 #1808 v0.39.8, by cammellos
💜 #1805 Add support for request messages by topics, by adambabik
💜 #1804 Peg clock value to whisper timestamp, by cammellos
💜 #1803 Split shhext into shhext and wakuext, by adambabik
💜 #1794 Nimbus backend initial implementation, by pedropombeiro
💜 #1775 faster tx fetching with less request, by rasom

---

COMMITS

Last week there were 14 commits.
🛠️ Fix wrong field being logged in EnvelopesMonitor by pedropombeiro
🛠️ bump to 0.40.0 by rasom
🛠️ status-im/status-react#9203 Faster tx fetching with less request *** How it worked before this PR on multiaccount creation: - On multiacc creation we scanned chain for eth and erc20 transfers. For each address of a new empty multiaccount this scan required 1. two eth_getBalance requests to find out that there is no any balance change between zero and the last block, for eth transfers 2. and chain-size/100000 (currently ~100) eth_getLogs requests, for erc20 transfers - For some reason we scanned an address of the chat account as well, and also accounts were not deduplicated. So even for an empty multiacc we scanned chain twice for each chat and main wallet addresses, in result app had to execute about 400 requests. - As mentioned above, eth_getBalance requests were used to check if there were any eth transfers, and that caused empty history in case if user already used all available eth (so that both zero and latest blocks show 0 eth for an address). There might have been transactions but we wouldn't fetch/show them. - There was no upper limit for the number of rpc requests during the scan, so it could require indefinite number of requests; the scanning algorithm was written so that we persisted the whole history of transactions or tried to scan form the beginning again in case of failure, giving up only after 10 minutes of failures. In result addresses with sufficient number of transactions would never be fully scanned and during these 10 minutes app could use gigabytes of internet data. - Failures were caused by eth_getBlockByNumber/eth_getBlockByHash requests. These requests return significantly bigger responses than eth_getBalance/eth_transactionsCount and it is likely that execution of thousands of them in parallel caused failures for accounts with hundreds of transactions. Even for an account with 12k we could successfully determine blocks with transaction in a few minutes using eth_getBalance requests, but eth_getBlock... couldn't be processed for this acc. - There was no caching for for eth_getBalance requests, and this caused in average 3-4 times more such requests than is needed. *** How it works now on multiaccount creation: - On multiacc creation we scan chain for last ~30 eth transactions and then check erc20 in the range where these eth transactions were found. For an empty address in multiacc this means: 1. two eth_getBalance transactions to determine that there was no balance change between zero and the last block; two eth_transactionsCount requests to determine there are no outgoing transactions for this address; total 4 requests for eth transfers 2. 20 eth_getLogs for erc20 transfers. This number can be lowered, but that's not a big deal - Deduplication of addresses is added and also we don't scan chat account, so a new multiacc requires ~25 (we also request latest block number and probably execute a few other calls) request to determine that multiacc is empty (comparing to ~400 before) - In case if address contains transactions we: 1. determine the range which contains 20-25 outgoing eth/erc20 transactions. This usually requires up to 10 eth_transactionCount requests 2. then we scan chain for eth transfers using eth_getBalance and eth_transactionCount (for double checking zero balances) 3. we make sure that we do not scan db for more than 30 blocks with transfers. That's important for accounts with mostly incoming transactions, because the range found on the first step might contain any number of incoming transfers, but only 20-25 outgoing transactions 4. when we found ~30 blocks in a given range, we update initial range from block using the oldest found block 5. and now we scan db for erc20transfers using eth_getLogs oldest-found-eth-block-latest-block, we make not more than 20 calls 6. when all blocks which contain incoming/outgoing transfers for a given address are found, we save these blocks to db and mark that transfers from these blocks are still to be fetched 7. Then we select latest ~30 (the number can be adjusted) blocks from these which were found and fetch transfers, this requires 3-4 requests per transfer. 8. we persist scanned range so that we know were to start next time 9. we dispatch an event which tells client that transactions are found 10. client fetches latest 20 transfers - when user presses "fetch more" button we check if app's db contains next 20 transfers, if not we scan chain again and return transfers after small fixes by rasom
🛠️ Fix ens names and contact (#1810) Currently the wrong pk format was used to query the in-memory contacts,
this commit corrects the behavior. by cammellos
🛠️ bump to 0.39.10 by adambabik
🛠️ Add support for request messages by topics (#1805) by adambabik
🛠️ bump to 0.39.9 by adambabik
🛠️ Split shhext into shhext and wakuext (#1803) by adambabik
🛠️ v0.39.8 (#1808) by cammellos
🛠️ Peg clock value to whisper timestamp (#1804) This commit pegs the clock value to maximum + 120 seconds from the whisper
timestamp.
In this way the we avoid the scenario where a client makes the timestamp
increase arbitrarely. by cammellos
🛠️ Working Nimbus branch with geth keystore by pedropombeiro
🛠️ In progress: Use Nimbus keystore by pedropombeiro
🛠️ Avoid passing node to subscriptions service by pedropombeiro
🛠️ Nimbus node support by pedropombeiro

---

CONTRIBUTORS

Last week there were 4 contributors.
👤 pedropombeiro
👤 rasom
👤 cammellos
👤 adambabik

---

STARGAZERS

Last week there were 2 stagazers.
⭐ jolestar
⭐ luoweisong
You all are the stars! 🌟

---

RELEASES

Last week there were no releases.

---

That's all for last week, please 👀 Watch and ⭐ Star the repository status-im/status-go to receive next weekly updates. 😃

You can also view all Weekly Digests by clicking here.

Your Weekly Digest bot. 📆