forked from pola-rs/polars
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
perf: Batch nested embed parquet decoding
This continues on pola-rs#17542 and now also batches the actual Nesting, not only the top decoder. I did the same benchmark with `POLARS_MAX_THREADS=1` and we again see a small speedup. But I suspect that the speedup is larger for `FixedSizeList`s and `Struct`s. ``` Benchmark 1: After Optimization Time (mean ± σ): 13.306 s ± 0.018 s [User: 12.661 s, System: 0.614 s] Range (min … max): 13.263 s … 13.335 s 10 runs Benchmark 2: Before Optimization Time (mean ± σ): 14.973 s ± 0.065 s [User: 14.422 s, System: 0.517 s] Range (min … max): 14.887 s … 15.129 s 10 runs Summary After Optimization ran 1.13 ± 0.01 times faster than Before Optimization ```
- Loading branch information
1 parent
a55d9bc
commit 0b157c5
Showing
5 changed files
with
136 additions
and
130 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.