-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use conf.useCompression config to decide if we should be compressing the cache [databricks] #4989
Conversation
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
...gin/src/main/311+-all/scala/com/nvidia/spark/rapids/shims/ParquetCachedBatchSerializer.scala
Outdated
Show resolved
Hide resolved
Has this been tested? I think the default compression type will also encode Snappy, so I wouldn't be surprised if this change had no effect as written. |
I tested this by |
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
I found a problem with cudf and created an issue for it in that repo here Converting this PR to draft until that bug is fixed |
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
build |
@jlowe do you have any more concerns? |
This PR makes PCBS to honor the
conf.useCompression
to decide if cache should be compressed using SNAPPYSigned-off-by: Raza Jafri rjafri@nvidia.com