Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update test_add_overflow_with_ansi_enabled and test_subtraction_overflow_with_ansi_enabled to check the exception type for Integral case. #6071

Conversation

HaoYang670
Copy link
Collaborator

@HaoYang670 HaoYang670 commented Jul 25, 2022

Signed-off-by: remzi 13716567376yh@gmail.com
This a subtask of #5196.

Changes in this PR.

Update test_add_overflow_with_ansi_enabled and test_subtraction_overflow_with_ansi_enabled to check the type of exception thrown.
Please note that we only update the IntegralType case and don't modify the DecimalType case, because it is difficult to get a Spark's Decimal from a cuDF's ColumnView, which makes it hard to use the RapidsErrorUtil.cannotChangeDecimalPrecisionError: https://github.com/NVIDIA/spark-rapids/blob/branch-22.08/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala#L1501-L1516

Signed-off-by: remzi <13716567376yh@gmail.com>
@HaoYang670
Copy link
Collaborator Author

build

@jlowe
Copy link
Member

jlowe commented Jul 25, 2022

Please note that we only update the IntegralType case and don't modify the DecimalType case, because it is difficult to get a Spark's Decimal from a cuDF's ColumnView, which makes it hard to use the RapidsErrorUtil.cannotChangeDecimalPrecisionError:

Why are we trying to get the Spark type from the cudf type? We know both the Spark type of the input and the Spark type of the output from the query plan. We may not know it in fixDecimalBounds given its limited parameters, but the code that eventually calls that method knows the input and output decimal types.

@jlowe
Copy link
Member

jlowe commented Jul 25, 2022

build

@jlowe jlowe added this to the July 22 - Aug 5 milestone Jul 25, 2022
@jlowe jlowe added the test Only impacts tests label Jul 25, 2022
@jlowe
Copy link
Member

jlowe commented Jul 25, 2022

build

1 similar comment
@jlowe
Copy link
Member

jlowe commented Jul 25, 2022

build

@jlowe
Copy link
Member

jlowe commented Jul 25, 2022

Hiccup on the artifact server, kicking the build again

@jlowe
Copy link
Member

jlowe commented Jul 25, 2022

build

@HaoYang670 HaoYang670 self-assigned this Jul 28, 2022
@HaoYang670
Copy link
Collaborator Author

We may not know it in fixDecimalBounds given its limited parameters, but the code that eventually calls that method knows the input and output decimal types.

The fixDecimalBounds method is called from here: https://github.com/NVIDIA/spark-rapids/blob/branch-22.08/sql-plugin/src/main/scala/com/nvidia/spark/rapids/decimalExpressions.scala#L45-L60.
If we want to get the decimal value, we should also pass input to the GpuCast.checkNFixDecimalBounds. Otherwise, we have to rebuild the GpuColumnVector from ColumnView in fixDecimalBounds, which is difficult.

As this would cause API changes or maybe more memory allocation, I'm not sure whether this is worth to do when our target is just to match the error message with Spark's.

@tgravescs
Copy link
Collaborator

build

@HaoYang670 HaoYang670 marked this pull request as draft August 1, 2022 06:43
@HaoYang670 HaoYang670 marked this pull request as ready for review August 1, 2022 07:50
@HaoYang670

This comment was marked as resolved.

Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to get the decimal value, we should also pass input to the GpuCast.checkNFixDecimalBounds.

Yep, that or separately pass the input DataType. Either works.

As this would cause API changes or maybe more memory allocation

We own the APIs in question (they are internal, not public), so this would not be a major change. There's no additional memory allocation involved to pass an existing object instance to a method, not sure what you're getting at there?

Determining the input datatype takes no additional memory, but besides the datatype we would need to extract a specific, offending value from the input to call cannotChangeDecimalPrecisionError which does require additional GPU processing and memory. As such I'm fine with this just matching the base exception type for now. We can put in the effort to extract a value in a followup PR if deemed necessary.

@sameerz sameerz merged commit e2d18fd into NVIDIA:branch-22.08 Aug 1, 2022
@HaoYang670 HaoYang670 deleted the update_test_add_subtraction_overflow_with_ansi_enabled branch August 1, 2022 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Only impacts tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants