Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Throw SparkDateTimeException for InvalidInput while casting in ANSI mode [databricks] #5731

Merged
merged 4 commits into from
Jun 6, 2022
Merged

Throw SparkDateTimeException for InvalidInput while casting in ANSI mode [databricks] #5731

merged 4 commits into from
Jun 6, 2022

Conversation

nartal1
Copy link
Collaborator

@nartal1 nartal1 commented Jun 3, 2022

This fixes #5719 .

Spark updated the error message to throw SparkDateTimeException in this PR: apache/spark#36591 .
Since SparkDateTimeException is a private class in org.apache.spark, I have included TrampolineUtil to call that Exception.
Current integration test passes now. No new test required.

@nartal1
Throw SparkDateTimeException for InvalidInput in CastToDateTime in AN…
ad907d8 
…SI mode

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@nartal1 nartal1 added the bug Something isn't working label Jun 3, 2022
@nartal1 nartal1 added this to the May 23 - Jun 3 milestone Jun 3, 2022
@nartal1 nartal1 self-assigned this Jun 3, 2022
@nartal1 nartal1 changed the title Throw SparkDateTimeException for InvalidInput in CastToDateTime in ANSI mode [databricks] Throw SparkDateTimeException for InvalidInput while casting in ANSI mode [databricks] Jun 3, 2022
@nartal1
Copy link
Collaborator Author

nartal1 commented Jun 3, 2022

build

@nartal1
add newline
f20f7f2 
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@nartal1
Copy link
Collaborator Author

nartal1 commented Jun 3, 2022

build

tgravescs
tgravescs reviewed Jun 3, 2022
View reviewed changes
Copy link
Collaborator

@tgravescs tgravescs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does the full error message look like now from us?

sql-plugin/src/main/330+/scala/com/nvidia/spark/rapids/shims/AnsiUtil.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/330+/scala/org/apache/spark/ShimTrampolineUtil.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/330+/scala/org/apache/spark/ShimTrampolineUtil.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/330+/scala/com/nvidia/spark/rapids/shims/AnsiUtil.scala Outdated Show resolved Hide resolved
@nartal1
addressed review comments
4c33ad4 
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@nartal1
Copy link
Collaborator Author

nartal1 commented Jun 3, 2022

what does the full error message look like now from us?

With the latest commit, the error message looks like this:

Input: Nan
org.apache.spark.SparkDateTimeException: The value NaN of the type DOUBLE cannot be cast to TIMESTAMP because it is malformed. Correct the value as per the syntax, or change its target type. To return NULL instead, use `try_cast`. If necessary set spark.sql.ansi.enabled to "false" to bypass this error.

Spark's error message:
org.apache.spark.SparkDateTimeException: The value NaN of the type "DOUBLE" cannot be cast to "TIMESTAMP" because it is malformed. Correct the value as per the syntax, or change its target type. To return NULL instead, use `try_cast`. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this erro

Input: +Infinity/ -Infinity.
We are throwing same error message for both positive and negative infinity where as Spark differentiates between those.
org.apache.spark.SparkDateTimeException: The value Infinity of the type DOUBLE cannot be cast to TIMESTAMP because it is malformed. Correct the value as per the syntax, or change its target type. To return NULL instead, use `try_cast`. If necessary set spark.sql.ansi.enabled to "false" to bypass this error.

@nartal1
addressed review comments
d17e121 
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@tgravescs
Copy link
Collaborator

build

gerashegalov
gerashegalov reviewed Jun 3, 2022
View reviewed changes
sql-plugin/src/main/330+/scala/com/nvidia/spark/rapids/shims/AnsiUtil.scala
s"NaN, Infinity or out-of-range values. To return NULL instead, use 'try_cast'. " +
s"If necessary set ${SQLConf.ANSI_ENABLED.key} to false to bypass this error."
val msg = s"The column contains out-of-range values. To return NULL instead, use " +
s"'try_cast'. If necessary set ${SQLConf.ANSI_ENABLED.key} to false to bypass this error."
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: in the comment it looks like the original Spark message uses quotes around the config key #5731 (comment)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created follow on issue : #5750 for changing all the error messages based on above format.

@HaoYang670
Copy link
Collaborator

Build failure is due to #5744

@sameerz
Copy link
Collaborator

sameerz commented Jun 5, 2022

build

@nartal1 nartal1 merged commit efb4d76 into NVIDIA:branch-22.06 Jun 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gerashegalov gerashegalov gerashegalov left review comments

@razajafri razajafri razajafri left review comments

@tgravescs tgravescs tgravescs approved these changes

Assignees

@nartal1 nartal1

Labels
bug Something isn't working
Projects
None yet
Milestone
May 23 - Jun 3
Development

Successfully merging this pull request may close these issues.

[BUG] test_cast_float_to_timestamp_ansi_for_nan_inf failed in spark330
6 participants
@nartal1 @tgravescs @HaoYang670 @sameerz @gerashegalov @razajafri