You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I just tried to run a test through the IDE. It failed with an error message
23/11/29 16:05:21.350 ScalaTest-run-running-ShufflePartitionerRetrySuite ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Multiple rapids-4-spark jars found in the classpath:
Please make sure there is only one rapids-4-spark jar in the classpath.
If it is impossible to fix the classpath you can suppress the error by setting spark.rapids.sql.allowMultipleJars to SAME_REVISION or ALWAYS.
But the problem is that spark.rapids.sql.allowMultipleJars is already set to SAME_REVISION. We need to have better error messages for depending on what is wrong. If there were no revisions found we should output an error saying that. If there were multiple different revisions found we should output an error message saying that and not suggest that they set an internal/non-documented config to something that is not going to work for them.
If we want users to be set this ever it needs to not be internal. If we don't want end users to ever set this without help from us, then we should not recommend how to fix it.
The text was updated successfully, but these errors were encountered:
To be clear I think that the case of zero revision numbers on the classpath is rather minor. It should not be common. But at the same time it is possible so if we see it we need to handle it properly. The one I am more concerned about is if there are multiple versions on the classpath and spark.rapids.sql.allowMultipleJars is set to SAME_REVISION. We are going to recommend that they set this hidden config to the value it is already set to? That feels off.
Describe the bug
I just tried to run a test through the IDE. It failed with an error message
But the problem is that
spark.rapids.sql.allowMultipleJars
is already set toSAME_REVISION
. We need to have better error messages for depending on what is wrong. If there were no revisions found we should output an error saying that. If there were multiple different revisions found we should output an error message saying that and not suggest that they set an internal/non-documented config to something that is not going to work for them.If we want users to be set this ever it needs to not be internal. If we don't want end users to ever set this without help from us, then we should not recommend how to fix it.
The text was updated successfully, but these errors were encountered: