Rust parse serialized #3444

arvidn · 2021-05-02T17:22:12Z

This is an attempt at speeding up Streamable deserialization of SerializedProgram. We need to half-parse the CLVM buffer to know the size of it.

The main part of this change is here: Chia-Network/clvm_rs#77

mariano54

Awesome! ill test it on my benchmark

arvidn · 2021-05-02T17:42:45Z

this can't land as-is. The setup.py commit needs to be updated and we need to cut a new clvm_rs release first too.

richardkiss · 2021-05-02T18:54:56Z

setup.py

@@ -6,7 +6,8 @@
    "chiabip158==1.0",  # bip158-style wallet filters
    "chiapos==1.0.1",  # proof of space
    "clvm==0.9.6",
-    "clvm_rs==0.1.6",
+    "maturin",
+    "clvm_rs@git+https://github.com/Chia-Network/clvm_rs.git@main#egg=clvm-rs",  # noqa


We can't do this in something going to pypi, unfortunately we have to do a real release of clvm_rs.

Oh, okay, you're aware.

richardkiss

Let's create a release of clvm_rs.

richardkiss · 2021-05-02T20:10:13Z

chia/types/blockchain_format/program.py

@@ -148,8 +148,8 @@ class SerializedProgram:

    @classmethod
    def parse(cls, f) -> "SerializedProgram":
-        tmp = sexp_buffer_from_stream(f)
-        return SerializedProgram.from_bytes(tmp)
+        length = serialized_length(f.getvalue()[f.tell() :])


It's possible that casting the bytes into a memoryview will save one copy of the blob. Might be worth checking to see how it benchmarks.

length = serialized_length(memoryview(f.getvalue())[f.tell():])

(I've been meaning to experiment with memoryview in python clvm to see if it makes for zero-cost substr.)

yeah, my initial idea was to use getbuffer(), but to feel confident with that API I would want to add more tests to the actual python bindings of clvm_rs too.

I agree that a longer term solution would be to take buffer views.

@Chida82

* 2 harvesting features (#3331) * 3 harvesting features: - Debug level shows the time for every quality lookup - Warning level if takes longer than 5 seconds - Allow configuration of plot loading interval (default 2 minutes) * Comment out super logging * Improve wallet consistency (#3305) * Improve wallet consistency * Improve CLI significantly, and fix self-tx balances * Fix await * Fix deadlock and test * Remove spam.sh * Changelog for 1.1.3 (#3345) * Changelog for 1.1.3 * minor updates * updates part 3 * Those engineers who don't update changelogs... :) * Apologies to @Chida82 who added log rotate count! (#3369) * Rust parse serialized (#3444) * use rust implementation for finding length of a serialized clvm program * bump clvm_rs version * Don't retry respond_peers message (#3508) Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com> Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com> Co-authored-by: Arvid Norberg <arvid@libtorrent.org>

@Chida82

* 2 harvesting features (#3331) * 3 harvesting features: - Debug level shows the time for every quality lookup - Warning level if takes longer than 5 seconds - Allow configuration of plot loading interval (default 2 minutes) * Comment out super logging * Improve wallet consistency (#3305) * Improve wallet consistency * Improve CLI significantly, and fix self-tx balances * Fix await * Fix deadlock and test * Remove spam.sh * Changelog for 1.1.3 (#3345) * Changelog for 1.1.3 * minor updates * updates part 3 * Those engineers who don't update changelogs... :) * Apologies to @Chida82 who added log rotate count! (#3369) * Rust parse serialized (#3444) * use rust implementation for finding length of a serialized clvm program * bump clvm_rs version * Don't retry respond_peers message (#3508) * don't increment counters for outgoing messages blocked by the rate limit. (#3518) This was causing a problem where outbound messages, blocked by the rate limiter, would still increment the counters as-if they had been sent. This, in turn, could cause other message types to get blocked becuase the rate limiter thought we had sent a lot of the other (blocked) message type. Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com> Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com> Co-authored-by: Arvid Norberg <arvid@libtorrent.org>

@Chida82

* 2 harvesting features (#3331) * 3 harvesting features: - Debug level shows the time for every quality lookup - Warning level if takes longer than 5 seconds - Allow configuration of plot loading interval (default 2 minutes) * Comment out super logging * Improve wallet consistency (#3305) * Improve wallet consistency * Improve CLI significantly, and fix self-tx balances * Fix await * Fix deadlock and test * Remove spam.sh * Changelog for 1.1.3 (#3345) * Changelog for 1.1.3 * minor updates * updates part 3 * Those engineers who don't update changelogs... :) * Apologies to @Chida82 who added log rotate count! (#3369) * Rust parse serialized (#3444) * use rust implementation for finding length of a serialized clvm program * bump clvm_rs version * Don't retry respond_peers message (#3508) * don't increment counters for outgoing messages blocked by the rate limit. (#3518) This was causing a problem where outbound messages, blocked by the rate limiter, would still increment the counters as-if they had been sent. This, in turn, could cause other message types to get blocked becuase the rate limiter thought we had sent a lot of the other (blocked) message type. * hide secret wallet key by default with 'chia keys show' (#3565) Co-authored-by: Adam Kelly <aqk@aqk.im> Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com> Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com> Co-authored-by: Arvid Norberg <arvid@libtorrent.org> Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com> Co-authored-by: Adam Kelly <aqk@aqk.im>

@Chida82

* 2 harvesting features (#3331) * 3 harvesting features: - Debug level shows the time for every quality lookup - Warning level if takes longer than 5 seconds - Allow configuration of plot loading interval (default 2 minutes) * Comment out super logging * Improve wallet consistency (#3305) * Improve wallet consistency * Improve CLI significantly, and fix self-tx balances * Fix await * Fix deadlock and test * Remove spam.sh * Changelog for 1.1.3 (#3345) * Changelog for 1.1.3 * minor updates * updates part 3 * Those engineers who don't update changelogs... :) * Apologies to @Chida82 who added log rotate count! (#3369) * Rust parse serialized (#3444) * use rust implementation for finding length of a serialized clvm program * bump clvm_rs version * Don't retry respond_peers message (#3508) * don't increment counters for outgoing messages blocked by the rate limit. (#3518) This was causing a problem where outbound messages, blocked by the rate limiter, would still increment the counters as-if they had been sent. This, in turn, could cause other message types to get blocked becuase the rate limiter thought we had sent a lot of the other (blocked) message type. * hide secret wallet key by default with 'chia keys show' (#3565) Co-authored-by: Adam Kelly <aqk@aqk.im> * brake if end of ses list (#3583) * brake if end of ses list * log * throw if wp peak is not heavier * comment * handle sync from scratch * comment Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com> Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com> Co-authored-by: Arvid Norberg <arvid@libtorrent.org> Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com> Co-authored-by: Adam Kelly <aqk@aqk.im> Co-authored-by: Almog De Paz <almogdepaz@gmail.com>

@Chida82

* 2 harvesting features (#3331) * 3 harvesting features: - Debug level shows the time for every quality lookup - Warning level if takes longer than 5 seconds - Allow configuration of plot loading interval (default 2 minutes) * Comment out super logging * Improve wallet consistency (#3305) * Improve wallet consistency * Improve CLI significantly, and fix self-tx balances * Fix await * Fix deadlock and test * Remove spam.sh * Changelog for 1.1.3 (#3345) * Changelog for 1.1.3 * minor updates * updates part 3 * Those engineers who don't update changelogs... :) * Apologies to @Chida82 who added log rotate count! (#3369) * Rust parse serialized (#3444) * use rust implementation for finding length of a serialized clvm program * bump clvm_rs version * Don't retry respond_peers message (#3508) * don't increment counters for outgoing messages blocked by the rate limit. (#3518) This was causing a problem where outbound messages, blocked by the rate limiter, would still increment the counters as-if they had been sent. This, in turn, could cause other message types to get blocked becuase the rate limiter thought we had sent a lot of the other (blocked) message type. * hide secret wallet key by default with 'chia keys show' (#3565) Co-authored-by: Adam Kelly <aqk@aqk.im> * brake if end of ses list (#3583) * brake if end of ses list * log * throw if wp peak is not heavier * comment * handle sync from scratch * comment * fix indentation issue in message send retry logic. factor out retry coroutine (#3629) * Fix test (#3350) * Mempool sorting and accept reverted pending transactions (#3683) * Sort by fee/cost, and fix pending tx issue in reorgs * Fix test name * Bring the seen list size back to normal. Co-authored-by: Mariano Sorgente <3069354+mariano54@users.noreply.github.com> Co-authored-by: Gene Hoffman <30377676+hoffmang9@users.noreply.github.com> Co-authored-by: Arvid Norberg <arvid@libtorrent.org> Co-authored-by: Adam Kelly <338792+aqk@users.noreply.github.com> Co-authored-by: Adam Kelly <aqk@aqk.im> Co-authored-by: Almog De Paz <almogdepaz@gmail.com>

mariano54 previously approved these changes May 2, 2021

View reviewed changes

arvidn dismissed mariano54’s stale review via abfe482 May 2, 2021 17:43

arvidn requested a review from richardkiss May 2, 2021 18:38

richardkiss reviewed May 2, 2021

View reviewed changes

richardkiss requested changes May 2, 2021

View reviewed changes

arvidn force-pushed the rust-parse-serialized branch from abfe482 to 513e5e1 Compare May 2, 2021 19:15

arvidn added 2 commits May 2, 2021 22:05

use rust implementation for finding length of a serialized clvm program

c13470c

bump clvm_rs version

4b0be19

arvidn force-pushed the rust-parse-serialized branch from 513e5e1 to 4b0be19 Compare May 2, 2021 20:06

richardkiss approved these changes May 2, 2021

View reviewed changes

richardkiss reviewed May 2, 2021

View reviewed changes

mariano54 merged commit ca4b252 into main May 3, 2021

mariano54 deleted the rust-parse-serialized branch May 3, 2021 09:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rust parse serialized #3444

Rust parse serialized #3444

arvidn commented May 2, 2021

mariano54 left a comment

arvidn commented May 2, 2021

richardkiss May 2, 2021

richardkiss May 2, 2021

richardkiss left a comment

richardkiss May 2, 2021

arvidn May 2, 2021

Rust parse serialized #3444

Rust parse serialized #3444

Conversation

arvidn commented May 2, 2021

mariano54 left a comment

Choose a reason for hiding this comment

arvidn commented May 2, 2021

richardkiss May 2, 2021

Choose a reason for hiding this comment

richardkiss May 2, 2021

Choose a reason for hiding this comment

richardkiss left a comment

Choose a reason for hiding this comment

richardkiss May 2, 2021

Choose a reason for hiding this comment

arvidn May 2, 2021

Choose a reason for hiding this comment