-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix tests failures in string_test.py #11030
Comments
The other tests all pass with ANSI mode disabled. |
This is odd. I can't seem to repro this failure now. |
I have double-checked my work. These tests don't fail. I'm closing this. We can reopen this if we see failures in the future. |
Yep, I think I spoke too soon. Reopening. |
The problem with I'm still investigating, but this is proving a time sink. |
Yep, this is still baffling. Here is the exception:
This is pointing into new code in Spark 4.0. Column {
UnresolvedFunction(Seq(name), inputs.map(_.expr), isDistinct, ignoreNulls = ignoreNulls)
} The complaint seems to be that I'm unable to repro this from the command line. Attaching a debugger allows this code to run through as well. This is occasionally reproducible from the pyspark shell. The exception is thrown from Spark CPU, and should not need the plugin for repro. I'm fairly confident that this is a bug in Spark 4, that routes |
As for the problem highlighted in
Edit: I have filed https://issues.apache.org/jira/browse/SPARK-48989 against Spark 4.x, to track the WholeStageCodeGen/NFE problem. This is happening on the CPU, without the plugin's involvement. |
Fixes NVIDIA#11030. This commit skips the tests pertaining to the following operations on Apache Spark 4.0: 1. Column.endswith() 2. substring_index() In both cases, the CPU version of Apache Spark 4.0 seems to cause exceptions, unrelated to the Spark RAPIDS plugin. See: 1. https://issues.apache.org/jira/browse/SPARK-48989 2. https://issues.apache.org/jira/browse/SPARK-48995 Signed-off-by: MithunR <mithunr@nvidia.com>
Fixes #11030. This commit skips the tests pertaining to the following operations on Apache Spark 4.0: 1. Column.endswith() 2. substring_index() In both cases, the CPU version of Apache Spark 4.0 seems to cause exceptions, unrelated to the Spark RAPIDS plugin. See: 1. https://issues.apache.org/jira/browse/SPARK-48989 2. https://issues.apache.org/jira/browse/SPARK-48995 Signed-off-by: MithunR <mithunr@nvidia.com>
The text was updated successfully, but these errors were encountered: