Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] SparkShimsImpl class initialization in SparkShimsSuite for 340 too eager #7676

Closed
gerashegalov opened this issue Feb 4, 2023 · 3 comments · Fixed by #8688
Closed
Labels
bug Something isn't working

Comments

@gerashegalov
Copy link
Collaborator

Describe the bug

*** RUN ABORTED ***
  java.lang.ExceptionInInitializerError:
  at com.nvidia.spark.rapids.shims.Spark340PlusShims.$init$(Spark340PlusShims.scala:33)
  at com.nvidia.spark.rapids.shims.SparkShimImpl$.<init>(SparkShims.scala:21)
  at com.nvidia.spark.rapids.shims.SparkShimImpl$.<clinit>(SparkShims.scala)
  at com.nvidia.spark.rapids.shims.spark340.SparkShimsSuite.$anonfun$new$1(SparkShimsSuite.scala:27)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
  ...
  Cause: java.lang.ClassNotFoundException: com.nvidia.spark.rapids.delta.DeltaProbeImpl
  at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
  at com.nvidia.spark.rapids.ShimLoader$.loadClass(ShimLoader.scala:333)
  at com.nvidia.spark.rapids.ShimLoader$.newInstanceOf(ShimLoader.scala:337)
  at com.nvidia.spark.rapids.ShimLoader$.newDeltaProbe(ShimLoader.scala:417)
  at com.nvidia.spark.rapids.delta.DeltaProvider$.provider$lzycompute(DeltaProvider.scala:46)
  at com.nvidia.spark.rapids.delta.DeltaProvider$.provider(DeltaProvider.scala:45)
  at com.nvidia.spark.rapids.delta.DeltaProvider$.apply(DeltaProvider.scala:49)
  at org.apache.spark.sql.rapids.ExternalSource$.deltaProvider$lzycompute(ExternalSource.scala:65)

Steps/Code to reproduce bug

mvn test -pl sql-plugin \
  -Dsuites=com.nvidia.spark.rapids.shims.spark340.SparkShimsSuite \
  -Dbuildver=340

Expected behavior
checking the shim version should not require Delta probing or tolerate CNF

Environment details (please complete the following information)

  • Environment location: local dev

Additional context
#7673

@gerashegalov gerashegalov added the bug Something isn't working label Feb 4, 2023
@firestarman
Copy link
Collaborator

firestarman commented Jul 3, 2023

Look like this has been fixed.

$ mvn test -pl sql-plugin   -Dsuites=com.nvidia.spark.rapids.shims.spark340.SparkShimsSuite   -Dbuildver=340
[INFO] --- scalatest-maven-plugin:2.0.2:test (test) @ rapids-4-spark-sql_2.12 ---
Run starting. Expected test count is: 2
com.nvidia.spark.rapids.shims.spark340.SparkShimsSuite:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
- spark shims version - https://github.com/NVIDIA/spark-rapids/issues/7676
- shuffle manager class
Run completed in 216 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
$ git diff
diff --git a/sql-plugin/src/test/spark340/scala/com/nvidia/spark/rapids/shims/spark340/SparkShimsSuite.scala b/sql-plugin/src/test/spark340/scala/com/nvidia/spark/rapids/shims/spark340/SparkShimsSuite.scala
index a216b273c..530791d5c 100644
--- a/sql-plugin/src/test/spark340/scala/com/nvidia/spark/rapids/shims/spark340/SparkShimsSuite.scala
+++ b/sql-plugin/src/test/spark340/scala/com/nvidia/spark/rapids/shims/spark340/SparkShimsSuite.scala
@@ -23,7 +23,7 @@ import com.nvidia.spark.rapids._
 import org.scalatest.funsuite.AnyFunSuite
 
 class SparkShimsSuite extends AnyFunSuite with FQSuiteName {
-  ignore("spark shims version - https://github.com/NVIDIA/spark-rapids/issues/7676") {
+  test("spark shims version - https://github.com/NVIDIA/spark-rapids/issues/7676") {
     assert(ShimLoader.getShimVersion === SparkShimVersion(3, 4, 0))
   }

@gerashegalov
Copy link
Collaborator Author

@firestarman why don't you put up this change as a PR?

@firestarman
Copy link
Collaborator

firestarman commented Jul 11, 2023

Just because I was not 100% sure about this even it can pass on my machine.
But I will do that and let our CI do more verification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants