Document Parquet and ORC compression support #550
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Sameer Raheja <sraheja@nvidia.com>
Signed-off-by: Sameer Raheja <sraheja@nvidia.com>
Signed-off-by: Sameer Raheja <sraheja@nvidia.com>
revans2
reviewed
Aug 12, 2020
docs/compatibility.md
Outdated
| @@ -184,6 +186,8 @@ Parquet will not be GPU-accelerated. If the INT96 timestamp format is not requir | |||
| compatibility with other tools then set `spark.sql.parquet.outputTimestampType` to | |||
| `TIMESTAMP_MICROS`. | |||
|
|
|||
| The plugin supports reading `uncompressed`, `snappy` and `gzip` Parquet files and writing `uncompressed` and `snappy` Parquet files. | |||
There was a problem hiding this comment.
I think the important thing we are missing is that we cannot tell what the file is compressed with until we start to read it and at that point we don't have the ability to fall back to the CPU so you will get an error. This is also true for ORC.
There was a problem hiding this comment.
Updated to note that we will error out on reading or writing an unsupported format
…format Signed-off-by: Sameer Raheja <sraheja@nvidia.com>
revans2
reviewed
Aug 12, 2020
docs/compatibility.md
Outdated
| @@ -164,6 +164,10 @@ similar issue exists for writing dates as described | |||
| appears to work for dates after the epoch as described | |||
| [here](https://github.com/NVIDIA/spark-rapids/issues/140). | |||
|
|
|||
| The plugin supports reading `uncompressed`, `snappy` and `zlib` ORC files and writing `uncompressed` | |||
| and `snappy` ORC files. At this point, the plugin does not have the ability to fall back to the | |||
| CPU when reading or writing an unsupported compression format, and will error out in that case. | |||
There was a problem hiding this comment.
Sorry it is just reading. Writing we will fall back to the CPU for an unsupported format.
There was a problem hiding this comment.
Corrected to only indicate reading will error out.
Signed-off-by: Sameer Raheja <sraheja@nvidia.com>
|
build |
revans2
approved these changes
Aug 12, 2020
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Compatibility notes for parquet and orc compression on read + write Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Fixing grammar Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Edit / rewrite Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Add note about erroring out in case of reading / writing unsupported format Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Correction - ony reading will error out. Signed-off-by: Sameer Raheja <sraheja@nvidia.com>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Compatibility notes for parquet and orc compression on read + write Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Fixing grammar Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Edit / rewrite Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Add note about erroring out in case of reading / writing unsupported format Signed-off-by: Sameer Raheja <sraheja@nvidia.com> * Correction - ony reading will error out. Signed-off-by: Sameer Raheja <sraheja@nvidia.com>
tgravescs
pushed a commit
to tgravescs/spark-rapids
that referenced
this pull request
Nov 30, 2023
…IDIA#550) Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com> Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
made.
Add documentation on compression formats supported for Parquet and ORC on read and write. Closes issue #341
added.
Reviewed in .md file in github.