-
Notifications
You must be signed in to change notification settings - Fork 891
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix null count in statistics for parquet #9303
Fix null count in statistics for parquet #9303
Conversation
Codecov Report
@@ Coverage Diff @@
## branch-21.12 #9303 +/- ##
================================================
- Coverage 10.79% 10.75% -0.04%
================================================
Files 116 116
Lines 18869 19482 +613
================================================
+ Hits 2036 2096 +60
- Misses 16833 17386 +553
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔥
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1 small q
Co-authored-by: Michael Wang <isVoid@users.noreply.github.com>
@gpucibot merge |
Fixes #9221
The bug was in the method for calculating nulls
(num_rows - num_valid)
which needednum_rows
to be summed intyped_statistics_chunk
because it's not part of the untyped struct. This caused a problem because the serialtyped_statistics_chunk.reduce()
had thenum_rows
summation but the parallelblock_reduce(typed_statistics_chunk)
did not.As part of the fix,
num_rows
is removed entirely andnull_count
is incremented in case of encountering nulls in the column.