Cast from numeric types to decimal type #1438

sperlingxx · 2020-12-31T09:21:33Z

This pull request is to support casting from numeric types to decimal type under GPU runtime, which is one of important feature of TPC-DS decimal support plan.

Here are limits of current implementation:

Has restriction on range of Float/Double values, because storage capacity of GPU decimal is restricted by INT64.
There may exists tiny difference between CPU results and GPU results when casting from floats/doubles. In CPU runtime, floats/doubles are firstly converted to strings, then converting to decimals from strings. We can't do the same thing under GPU runtime in current.

Signed-off-by: sperlingxx <lovedreamf@gmail.com>

sperlingxx · 2021-01-04T02:28:09Z

build

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/TypeChecks.scala

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

sperlingxx · 2021-01-05T13:25:13Z

build

sperlingxx · 2021-01-05T13:33:28Z

build

revans2 · 2021-01-05T14:38:49Z

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

@@ -372,6 +386,59 @@ case class GpuCast(
      case (ShortType | IntegerType | LongType | ByteType | StringType, BinaryType) =>
        input.getBase.asByteList(true)

+      case (IntegerType | LongType, dt: DecimalType) =>


What about byte and short which are also listed as supported for ANSI cast in the docs?

Sorry this is still missing byte. Is that expected?

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

docs/compatibility.md

sql-plugin/src/main/scala/com/nvidia/spark/rapids/TypeChecks.scala

sperlingxx · 2021-01-06T12:10:53Z

build

sperlingxx · 2021-01-06T12:28:12Z

build

sperlingxx · 2021-01-07T12:32:07Z

build

revans2

Just a few nits. Overall it looks really good

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala

sperlingxx · 2021-01-08T00:58:46Z

build

revans2 · 2021-01-08T16:19:16Z

shims/spark310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/Spark310Shims.scala

        import TypeSig._
        // nullChecks are the same

        override val booleanChecks: TypeSig = integral + fp + BOOLEAN + STRING
        override val sparkBooleanSig: TypeSig = numeric + BOOLEAN + STRING

-        override val integralChecks: TypeSig = integral + fp + BOOLEAN + STRING
+        override val integralChecks: TypeSig = numeric + BOOLEAN + TIMESTAMP + STRING + BINARY


Sorry this is a really minor nit. Spark does not support TIMESTAMP or BINARY (as you can see on the line below), so us claiming to support it is technically wrong. This should not impact the generated docs or the correctness of the code in any way.

It is also an issue for fpChecks with TIMESTAMP.

The only reason to fix it is to avoid confusion.

It's been fixed. My bad :)

revans2 · 2021-01-11T15:29:50Z

shims/spark310/src/main/scala/com/nvidia/spark/rapids/shims/spark310/Spark310Shims.scala

        import TypeSig._
        // nullChecks are the same

        override val booleanChecks: TypeSig = integral + fp + BOOLEAN + STRING
        override val sparkBooleanSig: TypeSig = numeric + BOOLEAN + STRING

-        override val integralChecks: TypeSig = integral + fp + BOOLEAN + STRING
+        override val integralChecks: TypeSig = numeric + STRING + STRING


BOOLEAN + STRING not STRING + STRING

sperlingxx · 2021-01-12T03:10:03Z

build

sperlingxx · 2021-01-12T06:26:05Z

build

pxLi · 2021-01-12T07:32:11Z

build

pxLi · 2021-01-12T08:09:48Z

build

Signed-off-by: sperlingxx <lovedreamf@gmail.com>

[auto-merge] bot-auto-merge-branch-23.10 to branch-23.12 [skip ci] [bot]

sperlingxx added 4 commits December 18, 2020 17:45

cache

85df224

support casting from numeric to decimal under GPU runtime

cd95e37

Signed-off-by: sperlingxx <lovedreamf@gmail.com>

Merge remote-tracking branch 'origin/branch-0.4' into cast_num_to_dec

cb18548

fix typo

56496cf

revans2 reviewed Jan 4, 2021

View reviewed changes

refine

a12aae7

refine

cb64ba0

revans2 changed the title ~~Cast from numeric types to decimal type~~ Ansi Cast from numeric types to decimal type Jan 5, 2021

revans2 reviewed Jan 5, 2021

View reviewed changes

Merge remote-tracking branch 'origin/branch-0.4' into cast_num_to_dec

e4e9fb9

gerashegalov reviewed Jan 6, 2021

View reviewed changes

docs/compatibility.md Outdated Show resolved Hide resolved

docs/compatibility.md Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/TypeChecks.scala Outdated Show resolved Hide resolved

refine

be08f68

sperlingxx added 2 commits January 6, 2021 20:26

fix typo

73d74d6

fix typo

2fc1dd6

sameerz linked an issue Jan 7, 2021 that may be closed by this pull request

[FEA] Support Decimal Casts #1330

Closed

sameerz added the feature request New feature or request label Jan 7, 2021

support nonAnsiCast

7e8914b

sperlingxx changed the title ~~Ansi Cast from numeric types to decimal type~~ Cast from numeric types to decimal type Jan 7, 2021

revans2 previously approved these changes Jan 7, 2021

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala Outdated Show resolved Hide resolved

sameerz mentioned this pull request Jan 8, 2021

[FEA] Support Decimal Casts #1330

Closed

tiny fix

b586cf9

sperlingxx dismissed revans2’s stale review via b586cf9 January 8, 2021 00:55

revans2 previously approved these changes Jan 8, 2021

View reviewed changes

sperlingxx added 2 commits January 11, 2021 10:38

Merge remote-tracking branch 'origin/branch-0.4' into cast_num_to_dec

9e7b38f

tiny fix

cbfe78f

sperlingxx dismissed revans2’s stale review via cbfe78f January 11, 2021 02:42

revans2 reviewed Jan 11, 2021

View reviewed changes

sperlingxx added 2 commits January 12, 2021 11:07

Merge remote-tracking branch 'origin/branch-0.4' into cast_num_to_dec

933f6de

tiny fix

d74f1cf

revans2 approved these changes Jan 12, 2021

View reviewed changes

revans2 merged commit 5b29df8 into NVIDIA:branch-0.4 Jan 12, 2021

sperlingxx deleted the cast_num_to_dec branch January 13, 2021 02:49

nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021

Cast from numeric types to decimal type (NVIDIA#1438)

3c23707

Signed-off-by: sperlingxx <lovedreamf@gmail.com>

nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021

Cast from numeric types to decimal type (NVIDIA#1438)

f34b010

Signed-off-by: sperlingxx <lovedreamf@gmail.com>

tgravescs pushed a commit to tgravescs/spark-rapids that referenced this pull request Nov 30, 2023

Merge pull request NVIDIA#1438 from NVIDIA/bot-auto-merge-branch-23.10

56a7e47

[auto-merge] bot-auto-merge-branch-23.10 to branch-23.12 [skip ci] [bot]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cast from numeric types to decimal type #1438

Cast from numeric types to decimal type #1438

sperlingxx commented Dec 31, 2020 •

edited

Loading

sperlingxx commented Jan 4, 2021

sperlingxx commented Jan 5, 2021

sperlingxx commented Jan 5, 2021

revans2 Jan 5, 2021

revans2 Jan 8, 2021

sperlingxx commented Jan 6, 2021

sperlingxx commented Jan 6, 2021

sperlingxx commented Jan 7, 2021

revans2 left a comment

sperlingxx commented Jan 8, 2021

revans2 Jan 8, 2021

sperlingxx Jan 11, 2021

revans2 Jan 11, 2021

sperlingxx Jan 12, 2021

sperlingxx commented Jan 12, 2021

sperlingxx commented Jan 12, 2021

pxLi commented Jan 12, 2021

pxLi commented Jan 12, 2021

Cast from numeric types to decimal type #1438

Cast from numeric types to decimal type #1438

Conversation

sperlingxx commented Dec 31, 2020 • edited Loading

sperlingxx commented Jan 4, 2021

sperlingxx commented Jan 5, 2021

sperlingxx commented Jan 5, 2021

revans2 Jan 5, 2021

Choose a reason for hiding this comment

revans2 Jan 8, 2021

Choose a reason for hiding this comment

sperlingxx commented Jan 6, 2021

sperlingxx commented Jan 6, 2021

sperlingxx commented Jan 7, 2021

revans2 left a comment

Choose a reason for hiding this comment

sperlingxx commented Jan 8, 2021

revans2 Jan 8, 2021

Choose a reason for hiding this comment

sperlingxx Jan 11, 2021

Choose a reason for hiding this comment

revans2 Jan 11, 2021

Choose a reason for hiding this comment

sperlingxx Jan 12, 2021

Choose a reason for hiding this comment

sperlingxx commented Jan 12, 2021

sperlingxx commented Jan 12, 2021

pxLi commented Jan 12, 2021

pxLi commented Jan 12, 2021

sperlingxx commented Dec 31, 2020 •

edited

Loading