-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix unit tests when AQE is enabled #558
Conversation
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
build |
tests/src/test/scala/com/nvidia/spark/rapids/BroadcastHashJoinSuite.scala
Outdated
Show resolved
Hide resolved
tests/src/test/scala/com/nvidia/spark/rapids/HashSortOptimizeSuite.scala
Outdated
Show resolved
Hide resolved
tests/src/test/scala/com/nvidia/spark/rapids/BroadcastHashJoinSuite.scala
Show resolved
Hide resolved
tests/src/test/scala/com/nvidia/spark/rapids/BroadcastHashJoinSuite.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Thanks @andygrove for reviewing. I think I have addressed all the review comments. Please take another look. |
tests/src/test/scala/com/nvidia/spark/rapids/BroadcastHashJoinSuite.scala
Outdated
Show resolved
Hide resolved
"GpuBuildLeft") | ||
assert(ShimLoader.getSparkShims.getBuildSide(res2.get.asInstanceOf[HashJoin]).toString == | ||
"GpuBuildRight") | ||
val initialPlan2 = plan2.queryExecution.executedPlan |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment here re removing intermediate variable
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@@ -35,13 +37,14 @@ class BroadcastHashJoinSuite extends SparkQueryCompareTestSuite { | |||
val df5 = df4.join(df3, Seq("longs"), "inner") | |||
|
|||
val plan = df5.queryExecution.executedPlan |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry, just one more nit. It would be better to get the reference to the executedPlan after calling collect rather than before. The goal is to get the final plan, so it would be clearer in this order. It isn't obvious to people looking this code that the executedPlan gets updated by the call to collect. Other than this, LGTM.
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
build |
* explicitly disable AQE in one test (NVIDIA#567) Signed-off-by: Andy Grove <andygrove@nvidia.com> * Enable using spark-submit to convert from csv to parquet (NVIDIA#565) Signed-off-by: Andy Grove <andygrove@nvidia.com> * xfail the Spark 3.1.0 integration tests that fail (NVIDIA#580) * xfail GpuTimeSub, arithmetic ops, and full outer join failures on 3.1.0 Signed-off-by: Thomas Graves <tgraves@nvidia.com> * xfail the rest of the 3.1.0 tests and enable 3.1.0 unit tests in the jenkins builds Signed-off-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Fix unit tests when AQE is enabled (NVIDIA#558) * Fix scala tests when AQE is enabled Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix broadcasthashjoin tests Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix indentation Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * Update buffer store to return compressed batches directly, add compression NVTX ranges (NVIDIA#572) * Update buffer store to return compressed batches directly, add compression NVTX ranges Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Update parameter name for clarity Signed-off-by: Jason Lowe <jlowe@nvidia.com> * xfail the tpch spark 3.1.0 tests that fail (NVIDIA#588) Signed-off-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Move GpuParquetScan/GpuOrcScan into Shim (NVIDIA#590) * Move GpuParquetScan to shim Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move scan overrides into shim Signed-off-by: Thomas Graves <tgraves@apache.org> * Rename GpuParquetScan object to match Signed-off-by: Thomas Graves <tgraves@apache.org> * Add tests for v2 datasources Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move OrcScan into shims Signed-off-by: Thomas Graves <tgraves@apache.org> * Fixes Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Fix imports Signed-off-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Filter nulls from joins where possible to improve performance. (NVIDIA#594) * Filter nulls from joins where possible to improve performance. Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Addressed review comments Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Updated patch for other shims * changelog generator Signed-off-by: Peixin Li <pxli@nyu.edu> * only filter out labels for issue * add nightly workflow on github actions Co-authored-by: Andy Grove <andygrove@users.noreply.github.com> Co-authored-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Niranjan Artal <50492963+nartal1@users.noreply.github.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com> Co-authored-by: Robert (Bobby) Evans <bobby@apache.org>
* explicitly disable AQE in one test (NVIDIA#567) Signed-off-by: Andy Grove <andygrove@nvidia.com> * Enable using spark-submit to convert from csv to parquet (NVIDIA#565) Signed-off-by: Andy Grove <andygrove@nvidia.com> * xfail the Spark 3.1.0 integration tests that fail (NVIDIA#580) * xfail GpuTimeSub, arithmetic ops, and full outer join failures on 3.1.0 Signed-off-by: Thomas Graves <tgraves@nvidia.com> * xfail the rest of the 3.1.0 tests and enable 3.1.0 unit tests in the jenkins builds Signed-off-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Fix unit tests when AQE is enabled (NVIDIA#558) * Fix scala tests when AQE is enabled Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix broadcasthashjoin tests Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix indentation Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * Update buffer store to return compressed batches directly, add compression NVTX ranges (NVIDIA#572) * Update buffer store to return compressed batches directly, add compression NVTX ranges Signed-off-by: Jason Lowe <jlowe@nvidia.com> * Update parameter name for clarity Signed-off-by: Jason Lowe <jlowe@nvidia.com> * xfail the tpch spark 3.1.0 tests that fail (NVIDIA#588) Signed-off-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Move GpuParquetScan/GpuOrcScan into Shim (NVIDIA#590) * Move GpuParquetScan to shim Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move scan overrides into shim Signed-off-by: Thomas Graves <tgraves@apache.org> * Rename GpuParquetScan object to match Signed-off-by: Thomas Graves <tgraves@apache.org> * Add tests for v2 datasources Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Move OrcScan into shims Signed-off-by: Thomas Graves <tgraves@apache.org> * Fixes Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Fix imports Signed-off-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com> * Filter nulls from joins where possible to improve performance. (NVIDIA#594) * Filter nulls from joins where possible to improve performance. Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Addressed review comments Signed-off-by: Robert (Bobby) Evans <bobby@apache.org> * Updated patch for other shims * changelog generator Signed-off-by: Peixin Li <pxli@nyu.edu> * only filter out labels for issue * add nightly workflow on github actions * fix format Co-authored-by: Andy Grove <andygrove@users.noreply.github.com> Co-authored-by: Thomas Graves <tgraves@apache.org> Co-authored-by: Thomas Graves <tgraves@nvidia.com> Co-authored-by: Niranjan Artal <50492963+nartal1@users.noreply.github.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com> Co-authored-by: Robert (Bobby) Evans <bobby@apache.org>
* Fix scala tests when AQE is enabled Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix broadcasthashjoin tests Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix indentation Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com>
* Fix scala tests when AQE is enabled Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix broadcasthashjoin tests Signed-off-by: Niranjan Artal <nartal@nvidia.com> * fix indentation Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com> * addressed review comments Signed-off-by: Niranjan Artal <nartal@nvidia.com>
…IDIA#558) Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com> Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
This fixes #452 and also BroadcastHashJoinSuite when AQE is enabled.