Add Databricks 13.3 shim boilerplate code and refactor Databricks 12.2 shim [databricks] #9510
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refactored DB 332 Shims and DB 341 Shims
sameerz
added
the
task
razajafri
force-pushed
the
SP-9175-db13.3-fix-compilation
branch
from
1aae690
to
c11f2a7
Compare
Signed-off-by: raza <rjafri@nvidia.com>
sql-plugin/src/main/spark332db/scala/com/nvidia/spark/rapids/shims/Spark332PlusDBShims.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/spark341db/scala/com/nvidia/spark/rapids/shims/GpuBatchScanExec.scala
Outdated
Show resolved
Hide resolved
.../src/main/spark341db/scala/org/apache/spark/sql/hive/rapids/shims/HiveProviderCmdShims.scala
Outdated
Show resolved
Hide resolved
jlowe
reviewed
Oct 24, 2023
sql-plugin/src/main/spark341db/scala/com/nvidia/spark/rapids/shims/GpuBatchScanExec.scala
Outdated
Show resolved
Hide resolved
gerashegalov
requested changes
Oct 24, 2023
...k341db/scala/org/apache/spark/sql/rapids/shims/spark341db/RapidsShuffleInternalManager.scala
Outdated
Show resolved
Hide resolved
...ugin/src/main/spark341db/scala/com/nvidia/spark/rapids/spark341db/RapidsShuffleManager.scala
Outdated
Show resolved
Hide resolved
...ugin/src/main/spark341db/scala/com/nvidia/spark/rapids/spark341db/RapidsShuffleManager.scala
Outdated
Show resolved
Hide resolved
|
build |
|
build |
jlowe
reviewed
Oct 26, 2023
sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/shims/GpuBatchScanExec.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/spark340/scala/com/nvidia/spark/rapids/shims/GpuBatchScanExec.scala
Outdated
Show resolved
Hide resolved
…ms/GpuBatchScanExec.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>
|
build |
…ms/GpuBatchScanExec.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com>
|
build |
jlowe
previously approved these changes
Oct 26, 2023
|
build |
sql-plugin/src/main/spark341db/scala/com/nvidia/spark/rapids/shims/Spark341PlusDBShims.scala
Show resolved
Hide resolved
sql-plugin/src/main/spark332db/scala/com/nvidia/spark/rapids/shims/shimExpressions.scala
Outdated
Show resolved
Hide resolved
firestarman
reviewed
Oct 27, 2023
sql-plugin/src/main/spark341db/scala/com/nvidia/spark/rapids/shims/Spark341PlusDBShims.scala
Outdated
Show resolved
Hide resolved
|
build |
|
@razajafri Please feel free to ping @pxLi and me when it's ready to add Databricks 13.3 in pre-merge and nightly CI. thanks! Or we may run Databricks 13.3 pre-merge CI first here regardless of IT failures to help check mvn build part? |
gerashegalov
changed the title
|
build |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces essential changes to enhance the compatibility and stability of our codebase, specifically focusing on the addition of Shim boilerplate code and the refactoring of existing shims from Databricks 12.2. These changes are crucial for ensuring seamless integration with Databricks 13.3
Changes Made:
Shim Boilerplate Code Addition: The PR includes the addition of necessary Shim boilerplate code, which includes
RapidsShuffleManager,SparkShimImplamongst othersShim Refactoring from Databricks 12.2: Some traits were added in Databricks 12.2 which are being reused in the new version. This PR refactors those traits into a separate file so it can be included with Databricks 13.3.
Other Refactoring: Some of the common functionality from Databricks 12.2 was refactored to a separate trait so it could be reused by Databricks 13.3.
Testing:
Tests will be addressed in future PRs