New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement ExistenceJoin Iterator using an auxiliary left semijoin #4796

Merged

gerashegalov merged 34 commits into NVIDIA:branch-22.04 from gerashegalov:gerashegalov/issue589-gathermap-as-an-existence-column

Mar 9, 2022

Collaborator

gerashegalov commented Feb 16, 2022 •

edited

Loading

This PR implements an iterator for ExistenceJoin

This PR computes ExistenceJoin by executing left semijoin via cuDF. The lhs GatherMap is scattering true into a Boolean column with all lhs.numRows being initiallyfalse . The rhs data is not gathered.
The PR also fixes regex matching against SparkPlan node strings. The previously used simple String mentions ExistenceJoin only in the CPU plan but does not print ExistenceJoin type as part of the Join exec string in the GPU plan.

Closes #589

Signed-off-by: Gera Shegalov gera@apache.org

gerashegalov added 7 commits

February 9, 2022 11:26

wip

a080f9a


          Merge remote-tracking branch 'origin/branch-22.04' into gerashegalov/…

cf135d7

…issue589-2


          Merge remote-tracking branch 'origin/branch-22.04' into gerashegalov/…

4ad4616

…issue589-2

wip

deba9a1

wip

664ba4e


          Use verboseStringWithSuffix(1000) for TreeNode match

a5da440

Signed-off-by: Gera Shegalov <gera@apache.org>


          Use Scala Int.MinValue

b55a3e1

Signed-off-by: Gera Shegalov <gera@apache.org>

Collaborator Author

gerashegalov commented Feb 16, 2022

build

gerashegalov self-assigned this

gerashegalov added the task label

gerashegalov added this to the Feb 14 - Feb 25 milestone


          config.md update for spark.rapids.sql.join.existence.enabled

6293dc1

Signed-off-by: Gera Shegalov <gera@apache.org>

Collaborator Author

gerashegalov commented Feb 16, 2022

build

gerashegalov mentioned this pull request

[FEA] Support ExistenceJoin #589

Closed

2 tasks

jlowe requested changes

View reviewed changes

build/buildall Outdated Show resolved Hide resolved

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

revans2 reviewed

View reviewed changes

sql-plugin/src/main/scala/com/nvidia/spark/rapids/JoinGatherer.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/JoinGatherer.scala Outdated Show resolved Hide resolved

gerashegalov added 6 commits

February 16, 2022 09:47


          Merge remote-tracking branch 'origin/branch-22.04' into gerashegalov/…

8c9597b

…issue589-gathermap-as-an-existence-column


          undo buildall changes

2517f73


          restore whitespace

193dbac


          existence join test with rhs duplicates

5ed19db


          semijoin-based implementation

1c332ec

Signed-off-by: Gera Shegalov <gera@apache.org>


          undo cosmetic change

a02992e

gerashegalov changed the title ~~Implement existence join gatherer on top of left outer GpuHashJoin~~ Implement existence join gatherer on top of left semijoin


          fix tagForGpu in GpuHashJoin

7e6bfcd

gerashegalov requested review from revans2 and jlowe

February 18, 2022 20:41

jlowe reviewed

View reviewed changes

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

integration_tests/src/main/python/join_test.py Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

revans2 reviewed

View reviewed changes

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated

+                  // cuDF executes left semijoin, the gatherer is constructed with a new
+                  // gather to gather every row from lhs
+                  //
+                  // we build a new rhs with a the "exists" Boolean column that has as many rows

Collaborator

revans2 Feb 18, 2022

This feels off. with a the "exists" I think the a is a typo

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated

+                  // semijoin lhs-GatherMap labeling rows that have at least one match in the original
+                  // rhs
+                  //
+                  val rhsExistsCB = withResource(Scalar.fromBool(false)) { falseScalar =>

Collaborator

revans2 Feb 18, 2022

This is way too deeply nested for me. Could we try to break it up some? The falseScalar is only used to create falseCV. It might also be nice to create a method for Table.scatter that takes the columnView and a single Scalar as input, and does all of the wrapping/unwrapping. to make this code that much more readable.

sql-plugin/src/main/scala/com/nvidia/spark/rapids/AbstractGpuJoinIterator.scala Outdated Show resolved Hide resolved

gerashegalov added 2 commits

February 18, 2022 17:00


          test updates

e333159


          draft

89c9a0e

gerashegalov added 4 commits

February 24, 2022 18:56


          mixedLeftSemiJoinGatherMap

29704bf


          mnemonic test ids

b19c98f


          refactoring

2948a66


          comment fix

6d55b6a

gerashegalov changed the title ~~[wip] Implement existence join gatherer on top of left semijoin~~ Implement existence join gatherer on top of left semijoin

gerashegalov marked this pull request as ready for review

February 25, 2022 17:16

gerashegalov requested review from jlowe and revans2

February 25, 2022 17:16

gerashegalov changed the title ~~Implement existence join gatherer on top of left semijoin~~ Implement ExistenceJoin Iterator using an auxiliary left semijoin

Collaborator Author

gerashegalov commented Feb 25, 2022

build

jlowe reviewed

View reviewed changes

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sameerz modified the milestones: Feb 14 - Feb 25, Feb 28 - Mar 18

gerashegalov added 5 commits

February 28, 2022 10:55

wip

a700f33


          broadcast hash join test

67d8616


          undo import py

e4c0a40


          undo explain

3a5a173


          undo explain

2f88244

gerashegalov requested a review from jlowe

March 2, 2022 06:31

jlowe reviewed

View reviewed changes

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Outdated Show resolved Hide resolved

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuHashJoin.scala Show resolved Hide resolved

integration_tests/src/main/python/join_test.py Outdated Show resolved Hide resolved

gerashegalov and others added 4 commits

March 2, 2022 10:46


          fixe bhj test id


          Update comment in join_test.py

02b037c

Co-authored-by: Jason Lowe <jlowe@nvidia.com>


          review comments

9f2ed60


          Merge remote-tracking branch 'origin/branch-22.04' into pr/gerashegal…

df4ad01

…ov/4796

gerashegalov requested a review from jlowe

March 4, 2022 06:40

Collaborator Author

gerashegalov commented Mar 4, 2022

build

1 similar comment

Collaborator

sameerz commented Mar 4, 2022

build

jlowe approved these changes

View reviewed changes

gerashegalov merged commit 98a731f into NVIDIA:branch-22.04

gerashegalov deleted the gerashegalov/issue589-gathermap-as-an-existence-column branch

March 9, 2022 02:02

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

task