Enable sort for single-level nesting struct columns on GPU #1883

gerashegalov · 2021-03-05T17:45:27Z

This PR contributes to #1605. It depends on rapidsai/cudf#7422

adds single-level struct type

Signed-off-by: Gera Shegalov gera@apache.org

Signed-off-by: Gera Shegalov <gera@apache.org>

…ct-sort

gerashegalov · 2021-03-05T18:09:14Z

Status:

TEST_PARALLEL=1 SPARK_HOME=~/spark-3.1.1-bin-hadoop3.2 integration_tests/run_pyspark_from_build.sh \
  -k "test_single_orderby and struct"

~~we observe the failure:~~ UPDATE: ✔️ resolved

ai.rapids.cudf.CudfException: cuDF failure at: /home/gshegalov/gits/cudf/cpp/src/sort/sort_column.cu:112: Column type must be relationally comparable
at ai.rapids.cudf.Table.orderBy(Native Method)
at ai.rapids.cudf.Table.orderBy(Table.java:1441)
at com.nvidia.spark.rapids.GpuSorter.$anonfun$sort$2(SortUtils.scala:183)

jlowe

What's here looks good, but there are some things missing.

At some point this needs to be built with mvn verify to generate the changes to supported_ops.md and those changes should be in this PR.

It would be nice to have some negative testing for this, i.e.: testing that we don't try to place sorts of deeply nested struct key columns or non-default struct sort ordering on the GPU.

…ct-sort

Signed-off-by: Gera Shegalov <gera@apache.org>

gerashegalov · 2021-03-11T09:13:10Z

build

gerashegalov · 2021-03-11T09:18:10Z

$ TEST_PARALLEL=1 COVERAGE_SUBMIT_FLAGS=$JDBSTR SPARK_HOME=~/spark-3.1.1-bin-hadoop3.2 integration_tests/run_pyspark_from_build.sh -k "test_single_struct and orderby"

=========================== short test summary info ============================
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS FIRST'>0-Struct(['child0', Struct(['child0', Byte],['child1', Short],['child2', Integer],['child3', Long],['child4', Float],['child5', Double],['child6', String],['child7', Boolean],['child8', Date],['child9', Timestamp],['child10', Null])])]
  second-level structs are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS FIRST'>0-Array(String)]
  arrays are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS FIRST'>0-Map(String(not_null),Map(String(not_null),String))]
  maps are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS FIRST'>1-Struct(['child0', Struct(['child0', Byte],['child1', Short],['child2', Integer],['child3', Long],['child4', Float],['child5', Double],['child6', String],['child7', Boolean],['child8', Date],['child9', Timestamp],['child10', Null])])]
  second-level structs are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS FIRST'>1-Array(String)]
  arrays are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS FIRST'>1-Map(String(not_null),Map(String(not_null),String))]
  maps are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS LAST'>-Struct(['child0', Byte],['child1', Short],['child2', Integer],['child3', Long],['child4', Float],['child5', Double],['child6', String],['child7', Boolean],['child8', Date],['child9', Timestamp],['child10', Null])]
  opposite null order not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS LAST'>-Struct(['child0', Struct(['child0', Byte],['child1', Short],['child2', Integer],['child3', Long],['child4', Float],['child5', Double],['child6', String],['child7', Boolean],['child8', Date],['child9', Timestamp],['child10', Null])])]
  opposite null order not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS LAST'>-Array(String)]
  opposite null order not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a ASC NULLS LAST'>-Map(String(not_null),Map(String(not_null),String))]
  opposite null order not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS LAST'>0-Struct(['child0', Struct(['child0', Byte],['child1', Short],['child2', Integer],['child3', Long],['child4', Float],['child5', Double],['child6', String],['child7', Boolean],['child8', Date],['child9', Timestamp],['child10', Null])])]
  second-level structs are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS LAST'>0-Array(String)]
  arrays are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS LAST'>0-Map(String(not_null),Map(String(not_null),String))]
  maps are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS FIRST'>-Struct(['child0', Byte],['child1', Short],['child2', Integer],['child3', Long],['child4', Float],['child5', Double],['child6', String],['child7', Boolean],['child8', Date],['child9', Timestamp],['child10', Null])]
  opposite null order not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS FIRST'>-Struct(['child0', Struct(['child0', Byte],['child1', Short],['child2', Integer],['child3', Long],['child4', Float],['child5', Double],['child6', String],['child7', Boolean],['child8', Date],['child9', Timestamp],['child10', Null])])]
  opposite null order not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS FIRST'>-Array(String)]
  opposite null order not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS FIRST'>-Map(String(not_null),Map(String(not_null),String))]
  opposite null order not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS LAST'>1-Struct(['child0', Struct(['child0', Byte],['child1', Short],['child2', Integer],['child3', Long],['child4', Float],['child5', Double],['child6', String],['child7', Boolean],['child8', Date],['child9', Timestamp],['child10', Null])])]
  second-level structs are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS LAST'>1-Array(String)]
  arrays are not supported
XFAIL ../../src/main/python/sort_test.py::test_single_nested_order_orderby[Column<'a DESC NULLS LAST'>1-Map(String(not_null),Map(String(not_null),String))]
  maps are not supported
=============== 4 passed, 4848 deselected, 20 xfailed in 43.43s ================

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/datetimeExpressions.scala

Signed-off-by: Gera Shegalov <gera@apache.org>

…ct-sort

- Create a single config for RAT in parent. - Update RAT exlusion list to filter out hidden diretories Signed-off-by: Gera Shegalov <gera@apache.org>

…ct-sort

Signed-off-by: Gera Shegalov <gera@apache.org>

revans2

GPU Sort uses upperbound, but not in all cases.

spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuSortExec.scala

Line 435 in ec2dcd7

withResource(sorter.upperBound(mergedBatch, cutoffCb)) { result =>

without rapidsai/cudf#7690 that means we have a ticking time bomb in the code.

Also the single partition work around makes no since to me at all. Until that is also fixed I see no way for us to merge this in.

gerashegalov · 2021-03-24T17:38:58Z

Also the single partition work around makes no since to me at all. Until that is also fixed I see no way for us to merge this in.

I understand the concern about GPU Sort.

As for the single partition workaround, It's just incremental delivery which mimics the progress we have on the cuDF side. It would make sense to me as long as it works as documented.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala

…ct-sort

gerashegalov · 2021-03-26T05:33:23Z

build

jlowe

This looks better, but I didn't see a test added for the ability to sort a table containing arrays but not using arrays as the key. Is that going to be done as a followup then?

gerashegalov · 2021-03-26T16:01:21Z

@jlowe these existing tests cover the scenarios you are calling out. they are passing now locally

gerashegalov · 2021-03-26T17:17:35Z

build

jlowe · 2021-03-26T17:44:17Z

these existing tests cover the scenarios you are calling out. they are passing now locally

Great, thanks @gerashegalov!

Adds single-level struct columns to sort. This PR contributes to NVIDIA#1605 The following limitations apply with this PR for a total sort, and will be resolved in follow-up PR's - only if the number of partitions is 1 - only if spark.rapids.sql.stableSort.enabled is true Signed-off-by: Gera Shegalov <gera@apache.org>

gerashegalov added 3 commits March 5, 2021 01:34

Enable sort of structs

cd3c436

Signed-off-by: Gera Shegalov <gera@apache.org>

struct gens for test_single_orderby

00ba3ff

Signed-off-by: Gera Shegalov <gera@apache.org>

Merge remote-tracking branch 'origin/branch-0.5' into issue-1605-stru…

1463028

…ct-sort

revert unneeded data_gen change

586f28f

jlowe added the SQL part of the SQL/Dataframe plugin label Mar 5, 2021

jlowe requested changes Mar 5, 2021

View reviewed changes

gerashegalov added 3 commits March 10, 2021 14:41

Merge remote-tracking branch 'origin/branch-0.5' into issue-1605-stru…

18d53a4

…ct-sort

interactive tests pass

c6407d0

Signed-off-by: Gera Shegalov <gera@apache.org>

add integration tests

88c8a9a

Signed-off-by: Gera Shegalov <gera@apache.org>

gerashegalov marked this pull request as ready for review March 11, 2021 09:14

gerashegalov requested review from GaryShen2008, NvTimLiu, revans2 and tgravescs as code owners March 11, 2021 09:14

gerashegalov changed the title ~~[WIP] Enable sort for single-level nesting struct columns on GPU~~ Enable sort for single-level nesting struct columns on GPU Mar 11, 2021

jlowe marked this pull request as draft March 11, 2021 15:23

jlowe reviewed Mar 11, 2021

View reviewed changes

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/datetimeExpressions.scala Outdated Show resolved Hide resolved

gerashegalov added 8 commits March 11, 2021 11:20

revert GpuTimeMath refactor

ddd60e4

Signed-off-by: Gera Shegalov <gera@apache.org>

Merge remote-tracking branch 'origin/branch-0.5' into issue-1605-stru…

ce7568c

…ct-sort

add tests for limit

ad934b2

Consolidate RAT settings in parent pom

5910d1a

- Create a single config for RAT in parent. - Update RAT exlusion list to filter out hidden diretories Signed-off-by: Gera Shegalov <gera@apache.org>

Merge branch 'ratPluginSettingsInParent' into issue-1605-struct-sort

c538bc2

Merge remote-tracking branch 'origin/branch-0.5' into issue-1605-stru…

d7ab1ea

…ct-sort

Enable saving data for debugging

33762de

Signed-off-by: Gera Shegalov <gera@apache.org>

Save schema for debugging

efbfa3f

Signed-off-by: Gera Shegalov <gera@apache.org>

improve error messages

196ebc9

revans2 requested changes Mar 24, 2021

View reviewed changes

gerashegalov added 2 commits March 25, 2021 01:14

correct CPU constraints

4b1f742

test fix

7e8c9f8

revans2 reviewed Mar 25, 2021

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala Outdated Show resolved Hide resolved

check for nesting in RP

9ee262b

gerashegalov requested review from revans2 and jlowe March 25, 2021 18:41

revans2 previously approved these changes Mar 25, 2021

View reviewed changes

jlowe reviewed Mar 25, 2021

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala Show resolved Hide resolved

gerashegalov added 3 commits March 25, 2021 17:52

wip

e5f9996

Merge remote-tracking branch 'origin/branch-0.5' into issue-1605-stru…

7d0bdcc

…ct-sort

reworked sortexec type list

5910092

gerashegalov dismissed revans2’s stale review via 5910092 March 26, 2021 02:50

jlowe approved these changes Mar 26, 2021

View reviewed changes

revans2 approved these changes Mar 26, 2021

View reviewed changes

gerashegalov merged commit 64f9d9b into NVIDIA:branch-0.5 Mar 26, 2021

gerashegalov deleted the issue-1605-struct-sort branch March 26, 2021 22:04

gerashegalov mentioned this pull request Apr 7, 2021

Remove restrictions in struct sort preventing calls to libcudf bounds #2092

Merged

sperlingxx mentioned this pull request Apr 25, 2021

[FEA] Support out of core sort on struct type keys #2257

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable sort for single-level nesting struct columns on GPU #1883

Enable sort for single-level nesting struct columns on GPU #1883

gerashegalov commented Mar 5, 2021 •

edited

Loading

gerashegalov commented Mar 5, 2021 •

edited

Loading

jlowe left a comment

gerashegalov commented Mar 11, 2021

gerashegalov commented Mar 11, 2021

revans2 left a comment

gerashegalov commented Mar 24, 2021

gerashegalov commented Mar 26, 2021

jlowe left a comment

gerashegalov commented Mar 26, 2021

gerashegalov commented Mar 26, 2021

jlowe commented Mar 26, 2021

Enable sort for single-level nesting struct columns on GPU #1883

Enable sort for single-level nesting struct columns on GPU #1883

Conversation

gerashegalov commented Mar 5, 2021 • edited Loading

gerashegalov commented Mar 5, 2021 • edited Loading

jlowe left a comment

Choose a reason for hiding this comment

gerashegalov commented Mar 11, 2021

gerashegalov commented Mar 11, 2021

revans2 left a comment

Choose a reason for hiding this comment

gerashegalov commented Mar 24, 2021

gerashegalov commented Mar 26, 2021

jlowe left a comment

Choose a reason for hiding this comment

gerashegalov commented Mar 26, 2021

gerashegalov commented Mar 26, 2021

jlowe commented Mar 26, 2021

gerashegalov commented Mar 5, 2021 •

edited

Loading

gerashegalov commented Mar 5, 2021 •

edited

Loading