Move GpuParquetScan/GpuOrcScan into Shim #590
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@apache.org>
|
build |
jlowe
approved these changes
Aug 19, 2020
pxLi
added a commit
to pxLi/spark-rapids
that referenced
this pull request
Aug 24, 2020
* explicitly disable AQE in one test (NVIDIA#567) Signed-off-by: Andy Grove <andygrove@nvidia.com> * Enable using spark-submit to convert from csv to parquet (NVIDIA#565) Signed-off-by: Andy Grove <andygrove@nvidia.com> * xfail the Spark 3.1.0 integration tests that fail (NVIDIA#580) * xfail GpuTimeSub, arithmetic ops, and full outer join failures on 3.1.0 Signed-off-by: Thomas Graves <tgraves@nvidia.com> * xfail the rest of the 3.1.0 tests and enable 3.1.0 unit tests in the jenkins builds Signed-off-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Fix unit tests when AQE is enabled (NVIDIA#558) * Fix scala tests when AQE is enabled Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix broadcasthashjoin tests Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix indentation Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * Update buffer store to return compressed batches directly, add compression NVTX ranges (NVIDIA#572) * Update buffer store to return compressed batches directly, add compression NVTX ranges Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Update parameter name for clarity Signed-off-by: Jason Lowe <jlowe@nvidia.com> * xfail the tpch spark 3.1.0 tests that fail (NVIDIA#588) Signed-off-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Move GpuParquetScan/GpuOrcScan into Shim (NVIDIA#590) * Move GpuParquetScan to shim Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move scan overrides into shim Signed-off-by: Thomas Graves <tgraves@apache.org> * Rename GpuParquetScan object to match Signed-off-by: Thomas Graves <tgraves@apache.org> * Add tests for v2 datasources Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move OrcScan into shims Signed-off-by: Thomas Graves <tgraves@apache.org> * Fixes Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Fix imports Signed-off-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Filter nulls from joins where possible to improve performance. (NVIDIA#594) * Filter nulls from joins where possible to improve performance. Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Addressed review comments Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Updated patch for other shims * changelog generator Signed-off-by: Peixin Li <pxli@nyu.edu> * only filter out labels for issue * add nightly workflow on github actions Co-authored-by: Andy Grove <andygrove@users.noreply.github.com> Co-authored-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Niranjan Artal <50492963+nartal1@users.noreply.github.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com> Co-authored-by: Robert (Bobby) Evans <bobby@apache.org>
pxLi
added a commit
to pxLi/spark-rapids
that referenced
this pull request
Aug 24, 2020
* explicitly disable AQE in one test (NVIDIA#567) Signed-off-by: Andy Grove <andygrove@nvidia.com> * Enable using spark-submit to convert from csv to parquet (NVIDIA#565) Signed-off-by: Andy Grove <andygrove@nvidia.com> * xfail the Spark 3.1.0 integration tests that fail (NVIDIA#580) * xfail GpuTimeSub, arithmetic ops, and full outer join failures on 3.1.0 Signed-off-by: Thomas Graves <tgraves@nvidia.com> * xfail the rest of the 3.1.0 tests and enable 3.1.0 unit tests in the jenkins builds Signed-off-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Fix unit tests when AQE is enabled (NVIDIA#558) * Fix scala tests when AQE is enabled Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix broadcasthashjoin tests Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix indentation Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * Update buffer store to return compressed batches directly, add compression NVTX ranges (NVIDIA#572) * Update buffer store to return compressed batches directly, add compression NVTX ranges Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Update parameter name for clarity Signed-off-by: Jason Lowe <jlowe@nvidia.com> * xfail the tpch spark 3.1.0 tests that fail (NVIDIA#588) Signed-off-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Move GpuParquetScan/GpuOrcScan into Shim (NVIDIA#590) * Move GpuParquetScan to shim Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move scan overrides into shim Signed-off-by: Thomas Graves <tgraves@apache.org> * Rename GpuParquetScan object to match Signed-off-by: Thomas Graves <tgraves@apache.org> * Add tests for v2 datasources Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move OrcScan into shims Signed-off-by: Thomas Graves <tgraves@apache.org> * Fixes Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Fix imports Signed-off-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Filter nulls from joins where possible to improve performance. (NVIDIA#594) * Filter nulls from joins where possible to improve performance. Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Addressed review comments Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Updated patch for other shims * changelog generator Signed-off-by: Peixin Li <pxli@nyu.edu> * only filter out labels for issue * add nightly workflow on github actions * fix format Co-authored-by: Andy Grove <andygrove@users.noreply.github.com> Co-authored-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Niranjan Artal <50492963+nartal1@users.noreply.github.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com> Co-authored-by: Robert (Bobby) Evans <bobby@apache.org>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Move GpuParquetScan to shim Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move scan overrides into shim Signed-off-by: Thomas Graves <tgraves@apache.org> * Rename GpuParquetScan object to match Signed-off-by: Thomas Graves <tgraves@apache.org> * Add tests for v2 datasources Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move OrcScan into shims Signed-off-by: Thomas Graves <tgraves@apache.org> * Fixes Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Fix imports Signed-off-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Move GpuParquetScan to shim Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move scan overrides into shim Signed-off-by: Thomas Graves <tgraves@apache.org> * Rename GpuParquetScan object to match Signed-off-by: Thomas Graves <tgraves@apache.org> * Add tests for v2 datasources Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move OrcScan into shims Signed-off-by: Thomas Graves <tgraves@apache.org> * Fixes Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Fix imports Signed-off-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com>
tgravescs
pushed a commit
to tgravescs/spark-rapids
that referenced
this pull request
Nov 30, 2023
…IDIA#590) Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com> Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
spark 3.1.0 FileScan changed which requires us to move our GpuParquetScan and GpuOrcScan into the Shim layer.
this also modifies the tests to run the readers against both Datasource v1 and Datasource v2 apis.
I choose to do this rather then pull in FileScan because this seems like less code and still allows us to pick up any changes to FileScan that happen in Spark.
closes #587