Create a shim for Spark 3.2.0 development #1704
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Gera Shegalov <gera@apache.org>
gerashegalov
requested review from
GaryShen2008,
jlowe,
NvTimLiu,
revans2 and
tgravescs
as code owners
jlowe
reviewed
Feb 10, 2021
tgravescs
reviewed
Feb 10, 2021
There was a problem hiding this comment.
we also need to update the docs and create RapidsShuffleManager.scala for 3.2.0.
I've been creating a RapidsShuffleManager.scala so that its easy for the user to set it to match the version, see:
https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-on-prem.html#enabling-rapidsshufflemanager
like:
Spark 3.1.0 (com.nvidia.spark.rapids.spark310.RapidsShuffleManager)
$ mvn verify -Pspark320tests -Dcuda.version=cuda11
$ find . -name scala-test-output.txt | xargs grep FAILED
./tests/target/surefire-reports/scala-test-output.txt:- IGNORE ORDER, WITH DECIMALS: short reduction aggs *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, WITH DECIMALS: float basic aggregates group by string literal *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, WITH DECIMALS: float basic aggregates group by float and string literal *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, WITH DECIMALS: float basic aggregates group by floats *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, WITH DECIMALS: float basic aggregates group by more_floats *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, NOT ON GPU[HashAggregateExec,AggregateExpression,AttributeReference,Alias,Literal,Min,Sum,Max,Average,Add,Multiply,Subtract,Cast,Count], WITH DECIMALS: partial on gpu: float basic aggregates group by more_floats *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, NOT ON GPU[HashAggregateExec,AggregateExpression,AttributeReference,Alias,Literal,Min,Sum,Max,Average,Add,Multiply,Subtract,Cast,Count,KnownFloatingPointNormalized,NormalizeNaNAndZero], WITH DECIMALS: final on gpu: float basic aggregates group by more_floats *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, WITH DECIMALS: nullable float basic aggregates group by more_floats *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, WITH DECIMALS: sum(floats) group by more_floats 2 partitions *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- INCOMPAT, IGNORE ORDER, WITH DECIMALS: empty df: reduction aggs *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- Test all supported casts with in-range values *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:- Join partitioned tables *** FAILED ***
./tests/target/surefire-reports/scala-test-output.txt:*** 12 TESTS FAILED *** |
|
build |
|
so I assume by the above output that means tests are failing, did you look why? these pass for you with spark311 right? |
tgravescs
reviewed
Feb 11, 2021
shims/spark320/src/main/scala/com/nvidia/spark/rapids/shims/spark320/Spark320Shims.scala
Show resolved
Hide resolved
tgravescs
reviewed
Feb 11, 2021
shims/spark320/src/main/scala/com/nvidia/spark/rapids/spark320/RapidsShuffleManager.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Gera Shegalov <gera@apache.org>
@tgravescs yes, |
|
Status ✔️ FIXED ✔️ : FIXED ✔️ : FIXED ✔️ : FIXED ✔️ : FIXED ✔️ : FIXED `INCOMPAT, IGNORE ORDER, WITH DECIMALS: float basic aggregates group by more_floats *** FAILED *** ✔️ : FIXED |
Signed-off-by: Gera Shegalov <gera@apache.org>
Signed-off-by: Gera Shegalov <gera@apache.org>
|
build |
gerashegalov
commented
Feb 28, 2021
| @@ -1857,7 +1857,7 @@ object GpuOverrides { | |||
| } | |||
| } | |||
|
|
|||
| override def convertToGpu(child: Expression): GpuExpression = GpuSum(child) | |||
| override def convertToGpu(child: Expression): GpuExpression = GpuSum(child, a.dataType) | |||
There was a problem hiding this comment.
this kind of change will be needed for avg for decimal support, the current tests are passing. Easiest way to identify expressions requiring this treatment in Spark:
➜ spark git:(master) ag 'def dataType: DataType = resultType'
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Sum.scala
47: override def dataType: DataType = resultType
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala
51: override def dataType: DataType = resultType
jlowe
reviewed
Mar 1, 2021
sql-plugin/src/main/java/com/nvidia/spark/rapids/GpuColumnVector.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Gera Shegalov <gera@apache.org>
Signed-off-by: Gera Shegalov <gera@apache.org>
Signed-off-by: Gera Shegalov <gera@apache.org>
jlowe
reviewed
Mar 1, 2021
...park311/src/main/scala/com/nvidia/spark/rapids/shims/spark311/SparkShimServiceProvider.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Gera Shegalov <gera@apache.org>
jlowe
requested changes
Mar 1, 2021
shims/spark320/src/main/scala/com/nvidia/spark/rapids/shims/spark320/Spark320Shims.scala
Show resolved
Hide resolved
tgravescs
reviewed
Mar 1, 2021
Signed-off-by: Gera Shegalov <gera@apache.org>
jlowe
approved these changes
Mar 1, 2021
|
build |
tgravescs
approved these changes
Mar 1, 2021
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
Signed-off-by: Gera Shegalov <gera@apache.org> Add a shim provider for Spark 3.2.0 development branch. Closes NVIDIA#1490 - fix overflows in aggregate buffer for GpuSum by wiring the explicit output column type - unit tests for the new shim - consolidate version profiles in the parent pom
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
Signed-off-by: Gera Shegalov <gera@apache.org> Add a shim provider for Spark 3.2.0 development branch. Closes NVIDIA#1490 - fix overflows in aggregate buffer for GpuSum by wiring the explicit output column type - unit tests for the new shim - consolidate version profiles in the parent pom
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Gera Shegalov gera@apache.org
Add a shim provider for Spark 3.2 development branch.
Splitting this from #1688 to reduce its scope
Closes #1490