Propagate local properties to broadcast execs #9610
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Haoyang Li <haoyangl@nvidia.com>
Signed-off-by: Haoyang Li <haoyangl@nvidia.com>
revans2
previously approved these changes
Nov 2, 2023
| data_gen = StructGen([['child'+str(ind), sub_gen] for ind, sub_gen in | ||
| enumerate(all_basic_gens + decimal_gens)], nullable=False) | ||
| enumerate(base_gens)], nullable=False) |
There was a problem hiding this comment.
Is there a reason this test changed? I get that we wanted to make this change for stack anyways. I'm just not sure combining it with an unrelated PR is a good thing.
There was a problem hiding this comment.
Ok got it, I will avoid combining unrelated things in one PR in the future. Thanks for the reminding.
|
Oops it looks like 312 and 313 do not like the code change. We might have to put in some shims to make this work properly. |
Signed-off-by: Haoyang Li <haoyangl@nvidia.com>
revans2
approved these changes
Nov 2, 2023
|
build |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #9464
This PR adds propagating local properties to
GpuBroadcastExchangeExecBaseandGpuSubqueryBroadcastExec.apache/spark@2f4a71252cc Added propagating local properties to subquery broadcast exec, because the local properties of subquery broadcast exec (which are usually propagated by broadcast exchange exec) to the broadcast threads would be wrong in some cases. And we have the same code in the plugin.
The test case in the commit message will fail because we didn't propagate local properties to
GpuBroadcastExchangeExecBaseso I added it. But after I adding it, the test case will pass no matter if the propagating local properties toGpuSubqueryBroadcastExecis added and no matter if I added theThread.sleepas the guide from the Spark commit.I personnally think the spark commit is reasonable and also applicable for plugin so I added it in this PR, but I haven't found a good way to test it.
As the spark commit message said, It is difficult to write a unit test that reproduces this behavior because usually
BroadcastExchangeExecis the first computing the broadcast variable.BTW,
GpuBroadcastToRowExecalready implements propagation, we may want to keep it andGpuSubqueryBroadcastExecconsistent if propagation is really unnecessary.spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastToRowExec.scala
Line 65 in 525c73e
This PR also fixes a nit from my previous commit.