Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement chunked Parquet reader #11867

Merged
merged 181 commits into from
Nov 18, 2022
Merged

Implement chunked Parquet reader #11867

merged 181 commits into from
Nov 18, 2022

Conversation

ttnghia
Copy link
Contributor

@ttnghia ttnghia commented Oct 5, 2022 •
edited
Loading

This adds chunked Parquet reader, which can perform chunked reading for accessing files by an iterative manner. Instead of reading the input file all at once, we can read it chunk by chunk, each chunk can be limited to be small enough to not exceed the cudf internal limit (2GB/2 billions rows):

auto reader = cudf::io::chunked_parquet_reader(byte_limit, read_opts);
do {
    auto const chunk = reader.read_chunk();
    // Process chunk
} while (reader.has_next());

nvdbaranec and others added 16 commits September 23, 2022 10:59
@nvdbaranec
Fix an issue where using num_rows and skip_rows on a parquet file con…
c2e8e87 
…taining a mix of nested and non-nested types would

result in incorrect row counts for the non-nested types. Also optimizes the preprocess path so that non-nested types
do not end up getting visited by the kernel.
@nvdbaranec
Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt
f330431
@nvdbaranec
Fixed an issue with the tests: input columns cannot have unsanitary l…
eadfd63 
…ists. Fixed an additional issue in the decoding where flat column types underneath

structs could end up ignoring skip_rows/num_rows.
@nvdbaranec
Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt
c4de038
@ttnghia
Copy parquet_reader_* into chunked_parquet_reader_*
222c9fe 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Modify chunked_parquet_reader_options
f49cfed 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Exploit inheritance to extend the options and options_builder classes
dd39804 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Remove unnecessary variable
81bc68f 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Misc
f8126be 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Add docs
0e7692c 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@nvdbaranec
PR feedback changes.
9f9eeb0
@nvdbaranec
Merge branch 'branch-22.12' into reader_preprocess_fix_and_opt
9b3ea62
@nvdbaranec
Fixed some compile errors from merging.
d2e409a
@ttnghia
Add chunked_parquet_reader
ed41ac1 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Add empty implementation
be782f2 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Merge branch 'branch-22.12' into parquet_reader
7908b66
@ttnghia ttnghia added feature request New feature or request 2 - In Progress Currently a work in progress libcudf Affects libcudf (C++/CUDA) code. Spark Functionality that helps Spark RAPIDS non-breaking Non-breaking change labels Oct 5, 2022
@ttnghia ttnghia self-assigned this Oct 5, 2022
@ttnghia ttnghia changed the title Implement chunked Parquet reader Implement chunked Parquet reader [skip ci] Oct 5, 2022
@ttnghia
Add a destructor and close
a7175c8 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Update docs
63a7bd6 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Fix comment
16c12d9 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Construct chunked_parquet_reader
cd85385 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Add comment
5944beb 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Rename function and implementing
7cfa72a 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Address some review comments
7203c67 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Merge branch 'branch-22.12' into parquet_reader
520448b
@ttnghia
Fix #endif
96eed8e 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>
@ttnghia
Merge branch 'branch-22.12' into parquet_reader
6697e3b 
Signed-off-by: Nghia Truong <nghiatruong.vn@gmail.com>

# Conflicts:
#	cpp/tests/CMakeLists.txt
@ttnghia ttnghia requested a review from vuule November 16, 2022 18:52
vuule
vuule reviewed Nov 16, 2022
View reviewed changes
Copy link
Contributor

@vuule vuule left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks solid, just a few more nitpick-y suggestions.
Awesome PR!

cpp/src/io/parquet/page_data.cu Outdated Show resolved Hide resolved
cpp/src/io/parquet/reader_impl_preprocess.cu Outdated Show resolved Hide resolved
cpp/src/io/parquet/reader_impl_preprocess.cu Outdated Show resolved Hide resolved
cpp/src/io/parquet/reader_impl_preprocess.cu Outdated Show resolved Hide resolved
cpp/src/io/parquet/reader_impl_preprocess.cu Outdated Show resolved Hide resolved
cpp/src/io/parquet/reader_impl_preprocess.cu Outdated Show resolved Hide resolved
cpp/src/io/parquet/reader_impl_preprocess.cu Outdated Show resolved Hide resolved
cpp/src/io/parquet/page_data.cu Outdated Show resolved Hide resolved
cpp/src/io/parquet/page_data.cu Outdated Show resolved Hide resolved
cpp/include/cudf/io/detail/parquet.hpp
Comment on lines +132 to +137
[[nodiscard]] bool has_next() const;

/**
* @copydoc cudf::io::chunked_parquet_reader::read_chunk
*/
[[nodiscard]] table_with_metadata read_chunk() const;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if read_chunk should be const but we can leave it for now.

@ttnghia ttnghia mentioned this pull request Nov 17, 2022
Merged
vuule
vuule approved these changes Nov 17, 2022
View reviewed changes
Copy link
Contributor

@vuule vuule left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing most of the nitpicks!
We can address some remaining (riskier) clean up ideas in separate PRs.

@ttnghia
Copy link
Contributor Author

ttnghia commented Nov 17, 2022

@gpucibot merge

1 similar comment
@nvdbaranec
Copy link
Contributor

@gpucibot merge

@rapids-bot rapids-bot bot merged commit 3fb09d1 into rapidsai:branch-22.12 Nov 18, 2022
@ttnghia ttnghia deleted the parquet_reader branch November 18, 2022 03:05
ajschmidt8 pushed a commit that referenced this pull request Nov 18, 2022
@ttnghia
Implement JNI for chunked Parquet reader (#11961)
782fba3 
This adds JNI for chunked Parquet reader. It depends on the chunked Parquet reader implementation PR  (#11867).

Authors:
   - https://github.com/nvdbaranec
   - Nghia Truong (https://github.com/ttnghia)

Approvers:
   - MithunR (https://github.com/mythrocks)
   - Robert (Bobby) Evans (https://github.com/revans2)
@mythrocks mythrocks mentioned this pull request Nov 21, 2022
Merged
@revans2 revans2 mentioned this pull request Nov 22, 2022
Closed
@vuule vuule mentioned this pull request Dec 6, 2022
Closed
@etseidl etseidl mentioned this pull request Mar 17, 2023
Closed
3 tasks
This was referenced Sep 10, 2023
Closed
Open
@GregoryKimball GregoryKimball mentioned this pull request Feb 26, 2024
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nvdbaranec nvdbaranec nvdbaranec left review comments

@PointKernel PointKernel PointKernel approved these changes

@vuule vuule vuule approved these changes

@elstehle elstehle Awaiting requested review from elstehle elstehle is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

@davidwendt davidwendt Awaiting requested review from davidwendt davidwendt is a code owner automatically assigned from rapidsai/cudf-cpp-codeowners

Assignees

@ttnghia ttnghia

@nvdbaranec nvdbaranec

Labels
3 - Ready for Review Ready for review by team CMake CMake build issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change Spark Functionality that helps Spark RAPIDS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Split batches from parquet that are too large, and try to guess better before decompressing
5 participants
@ttnghia @davidwendt @nvdbaranec @PointKernel @vuule