Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Concat() Exception bug fix #11039

Merged
merged 6 commits into from
Jun 13, 2024
Merged

Concat() Exception bug fix #11039

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3de33d1
concat_null_bug_fix
Feng-Jiang28 Jun 11, 2024
9ec889e
concat_null_bug_fix
Feng-Jiang28 Jun 11, 2024
af48cbb
Setting modified
Feng-Jiang28 Jun 11, 2024
7de0b15
Merge remote-tracking branch 'upstream/branch-24.08' into concat_bug_fix
Feng-Jiang28 Jun 11, 2024
c6feba1
remove comment
Feng-Jiang28 Jun 11, 2024
eb09724
concat considered as empty string
Feng-Jiang28 Jun 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions sql-plugin/src/main/scala/org/apache/spark/sql/rapids/collectionOperations.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ case class GpuConcat(children: Seq[Expression]) extends GpuComplexTypeMergingExp

override def columnarEval(batch: ColumnarBatch): GpuColumnVector = {
val res = dataType match {
// Explicitly return null for empty concat as Spark, since cuDF doesn't support empty concat.
case dt if children.isEmpty => GpuScalar.from(null, dt)
// in Spark concat() will be considered as an empty string here
case dt if children.isEmpty => GpuScalar("", dt)
// For single column concat, we pass the result of child node to avoid extra cuDF call.
case _ if children.length == 1 => children.head.columnarEval(batch)
case StringType => stringConcat(batch)
Expand Down
1 change: 0 additions & 1 deletion tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,6 @@ class RapidsTestSettings extends BackendTestSettings {
enableSuite[RapidsMathFunctionsSuite]
enableSuite[RapidsRegexpExpressionsSuite]
enableSuite[RapidsStringExpressionsSuite]
.exclude("concat", KNOWN_ISSUE("https://github.com/NVIDIA/spark-rapids/issues/10775"))
.exclude("string substring_index function", KNOWN_ISSUE("https://github.com/NVIDIA/spark-rapids/issues/10775"))
.exclude("SPARK-22498: Concat should not generate codes beyond 64KB", KNOWN_ISSUE("https://github.com/NVIDIA/spark-rapids/issues/10775"))
.exclude("SPARK-22549: ConcatWs should not generate codes beyond 64KB", KNOWN_ISSUE("https://github.com/NVIDIA/spark-rapids/issues/10775"))
Expand Down