Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use nvCOMP for ZLIB decompression in ORC reader #11024

Merged
merged 1 commit into from
Jun 2, 2022
Merged

Use nvCOMP for ZLIB decompression in ORC reader #11024

merged 1 commit into from
Jun 2, 2022

Conversation

vuule
Copy link
Contributor

@vuule vuule commented Jun 1, 2022 •
edited
Loading

Issue #11023

Adds DEFLATE compression type to the nvCOMP adapter.
ORC reader uses the adapter when nvCOMP experimental integrations are enabled (for now). Otherwise behavior is unchanged and the internal implementation is still used.
No tests as no visible behavior change is expected.

Pending: Performance impact

@github-actions github-actions bot added the libcudf Affects libcudf (C++/CUDA) code. label Jun 1, 2022
@vuule vuule added feature request New feature or request cuIO cuIO issue non-breaking Non-breaking change labels Jun 1, 2022
@vuule vuule self-assigned this Jun 1, 2022
@codecov
Copy link

codecov bot commented Jun 2, 2022 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.08@7f359e0). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             branch-22.08   #11024   +/-   ##
===============================================
  Coverage                ?   86.34%           
===============================================
  Files                   ?      144           
  Lines                   ?    22710           
  Branches                ?        0           
===============================================
  Hits                    ?    19608           
  Misses                  ?     3102           
  Partials                ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f359e0...8708673. Read the comment docs.

@vuule vuule marked this pull request as ready for review June 2, 2022 04:11
@vuule vuule requested a review from a team as a code owner June 2, 2022 04:11
@vuule vuule requested review from trxcllnt and davidwendt June 2, 2022 04:11
@vuule
Copy link
Contributor Author

vuule commented Jun 2, 2022

Perf impact evaluation blocked on ZLIB compression issues in the writer. We can merge this PR without the perf evaluation, but will need to run the benchmarks before enabling this feature by default.

jbrennan333
jbrennan333 approved these changes Jun 2, 2022
View reviewed changes
Copy link
Contributor

@jbrennan333 jbrennan333 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 this looks good to me.

bdice
bdice approved these changes Jun 2, 2022
View reviewed changes
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good to me!

cpp/src/io/comp/nvcomp_adapter.cpp
@@ -29,6 +29,14 @@
#define NVCOMP_HAS_ZSTD 0
#endif

#define NVCOMP_DEFLATE_HEADER <nvcomp/deflate.h>
#if __has_include(NVCOMP_DEFLATE_HEADER)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. This is the first time I've seen __has_include. It's a standardized feature since C++17.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, when I was adding ZSTD support, I searched libcudf for existing uses and found none :D
I'm not sure if we covered it in the C++17 talks we did a while ago.

@vuule
Copy link
Contributor Author

vuule commented Jun 2, 2022

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 447028f into rapidsai:branch-22.08 Jun 2, 2022
@vuule vuule deleted the fea-nvcomp-inflate branch June 2, 2022 22:24
@vuule
Copy link
Contributor Author

vuule commented Jun 3, 2022

Managed to get the writer ZLIB compression up and running.
ORC reader benchmark does not show a significant difference between cuIO and nvCOMP ZLIB decompression performance. Throughput is up 2% on average with nvCOMP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bdice bdice bdice approved these changes

@ttnghia ttnghia ttnghia approved these changes

@davidwendt davidwendt davidwendt approved these changes

@jbrennan333 jbrennan333 jbrennan333 approved these changes

@trxcllnt trxcllnt Awaiting requested review from trxcllnt trxcllnt is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@vuule vuule

Labels
cuIO cuIO issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@vuule @bdice @ttnghia @davidwendt @jbrennan333