-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: Faster bitpacking for Parquet writer #16278
Conversation
No need to keep benchmark in code base
a86ec6f
to
dc1b9a2
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #16278 +/- ##
==========================================
+ Coverage 80.80% 81.34% +0.53%
==========================================
Files 1393 1403 +10
Lines 179406 183257 +3851
Branches 2921 2922 +1
==========================================
+ Hits 144971 149063 +4092
+ Misses 33932 33691 -241
Partials 503 503 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice improvement! 👏
I replaced the Parquet bit-packing algorithm with a modified version of the scalar algorithm from https://github.com/quickwit-oss/bitpacking (unsure where/how to give credit)
Can you a comment saying that you adapted from .
b33e69b
to
4a5734a
Compare
Nice improvement! Thanks a lot @thalassemia. 🙌 Hope you find some on the reading side as well. ;) |
I replaced the Parquet bit-packing algorithm with a modified version of the scalar algorithm from https://github.com/quickwit-oss/bitpacking (unsure where/how to give credit). I ensured the new algorithm works by adding unit tests that encode/decode random data.
I added a microbenchmark in my first commit to help measure the performance gain.
I removed the microbenchmark in a later commit because it requires nightly Rust and is probably not useful outside of this PR.
Other miscellaneous fixes: