Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Error message for multiple jars on classpath is wrong #9897

Closed
revans2 opened this issue Nov 29, 2023 · 2 comments · Fixed by #9922
Closed

[BUG] Error message for multiple jars on classpath is wrong #9897

revans2 opened this issue Nov 29, 2023 · 2 comments · Fixed by #9922
Assignees
Labels
bug Something isn't working

Comments

@revans2
Copy link
Collaborator

revans2 commented Nov 29, 2023

Describe the bug
I just tried to run a test through the IDE. It failed with an error message

23/11/29 16:05:21.350 ScalaTest-run-running-ShufflePartitionerRetrySuite ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Multiple rapids-4-spark jars found in the classpath:

Please make sure there is only one rapids-4-spark jar in the classpath.
If it is impossible to fix the classpath you can suppress the error by setting spark.rapids.sql.allowMultipleJars to SAME_REVISION or ALWAYS.

But the problem is that spark.rapids.sql.allowMultipleJars is already set to SAME_REVISION. We need to have better error messages for depending on what is wrong. If there were no revisions found we should output an error saying that. If there were multiple different revisions found we should output an error message saying that and not suggest that they set an internal/non-documented config to something that is not going to work for them.

If we want users to be set this ever it needs to not be internal. If we don't want end users to ever set this without help from us, then we should not recommend how to fix it.

@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Nov 29, 2023
@thirtiseven thirtiseven self-assigned this Nov 30, 2023
@thirtiseven
Copy link
Collaborator

thirtiseven commented Nov 30, 2023

Hi @revans2 may I know how you build the project and test it? Oh sorry it is running a test through the IDE, now I understand

@revans2
Copy link
Collaborator Author

revans2 commented Nov 30, 2023

To be clear I think that the case of zero revision numbers on the classpath is rather minor. It should not be common. But at the same time it is possible so if we see it we need to handle it properly. The one I am more concerned about is if there are multiple versions on the classpath and spark.rapids.sql.allowMultipleJars is set to SAME_REVISION. We are going to recommend that they set this hidden config to the value it is already set to? That feels off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants