Chunking input before writing a ParquetCachedBatch #1265
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
razajafri
requested review from
GaryShen2008,
jlowe,
NvTimLiu,
revans2 and
tgravescs
as code owners
razajafri
commented
Dec 4, 2020
| @@ -253,6 +253,16 @@ private case class CloseableColumnBatchIterator(iter: Iterator[ColumnarBatch]) e | |||
| * This class assumes, the data is Columnar and the plugin is on | |||
| */ | |||
| class ParquetCachedBatchSerializer extends CachedBatchSerializer with Arm { | |||
|
|
|||
There was a problem hiding this comment.
This change was made to gain access to the Inner class, open to suggestions to make this better
|
This is pretty much done. It's still a WIP because the thing remaining is that the tests won't compile with Spark 3.0.0. I have used reflection for some tests that were simple but for others, especially the ones on the CPU, the tests looked really hard to read after I tried reflection. @tgravescs following up on our conversation, I think we should think about alternatives like you said, maybe these tests can live with the shim310? |
This was referenced Dec 4, 2020
Closed
|
As decided in the sprint-planning I am removing the WIP tag and marking this for review. We will include tests for the CPU part when we have a framework to compile tests selectively |
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
This commit should be reverted once we have a framework for selectively building tests based on Spark version
razajafri
force-pushed
the
chunking_input
branch
from
a4835aa
to
0f817e2
Compare
razajafri
changed the title
jlowe
requested changes
Dec 4, 2020
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Show resolved
Hide resolved
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Outdated
Show resolved
Hide resolved
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Show resolved
Hide resolved
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Show resolved
Hide resolved
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Show resolved
Hide resolved
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Show resolved
Hide resolved
tests/src/test/scala/com/nvidia/spark/rapids/ParquetWriterSuite.scala
Outdated
Show resolved
Hide resolved
This was referenced Dec 5, 2020
Closed
…ark310/ParquetCachedBatchSerializer.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>
jlowe
previously approved these changes
Dec 5, 2020
|
build |
1 similar comment
|
build |
|
build |
1 similar comment
|
build |
|
build |
1 similar comment
|
build |
|
Blossom is timing out. Kicking it off again |
|
Build |
1 similar comment
|
Build |
revans2
reviewed
Dec 7, 2020
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Show resolved
Hide resolved
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Show resolved
Hide resolved
...310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala
Outdated
Show resolved
Hide resolved
revans2
previously approved these changes
Dec 7, 2020
|
build |
revans2
approved these changes
Dec 7, 2020
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Chunking input Signed-off-by: Raza Jafri <rjafri@nvidia.com> * improved tests * updated * rearranged imports * renamed the tests * REVERT THIS COMMIT This commit should be reverted once we have a framework for selectively building tests based on Spark version * Update shims/spark310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * build failure * empty commit to kick off ci * fix for building against Spark 3.0.0 * addressed review comments * addressed review comments Co-authored-by: Raza Jafri <rjafri@nvidia.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Chunking input Signed-off-by: Raza Jafri <rjafri@nvidia.com> * improved tests * updated * rearranged imports * renamed the tests * REVERT THIS COMMIT This commit should be reverted once we have a framework for selectively building tests based on Spark version * Update shims/spark310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/ParquetCachedBatchSerializer.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * build failure * empty commit to kick off ci * fix for building against Spark 3.0.0 * addressed review comments * addressed review comments Co-authored-by: Raza Jafri <rjafri@nvidia.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com>
tgravescs
pushed a commit
to tgravescs/spark-rapids
that referenced
this pull request
Nov 30, 2023
…IDIA#1265) Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR deals with breaking up the incoming batches before writing them to a ParquetCachedBatch to make sure the CachedBuffer never exceeds the Int max-value.
I am still working on writing the tests to test the CPU writing. This PR so far only has tests for the
compressColumnarBatchWithParquet