Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Implement shuffle optimization when AQE is enabled #493

Closed
andygrove opened this issue Jul 31, 2020 · 2 comments
Closed

[FEA] Implement shuffle optimization when AQE is enabled #493

andygrove opened this issue Jul 31, 2020 · 2 comments
Assignees
Labels
feature request New feature or request

Comments

@andygrove
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Once we merge #462 we will lose the optimization of wrapping a GpuShuffleExchangeExec in a GpuCoalesceBatches, which we have when AQE is off.

We should explore ways to re-enable this, even though we are not seeing a performance degredation from the TPCxBB benchmarks. It is possible that other AQE optimizations are offseting the impact of this.

Describe the solution you'd like
We have to return something that implements ShuffleExchangeLike when AQE creates new shuffle query stages, so we can't return GpuCoalesceBatchesExec.

We could potentially create some kind of wrapper for GpuCoalesceBatchesExec(GpuShuffleExchangeExec) that implements ShuffleExchangeLike. Another option would be to build the coalesce functionality directly into GpuShuffleExchangeExec.

Describe alternatives you've considered
None.

Additional context
None.

@andygrove andygrove added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jul 31, 2020
@andygrove andygrove added this to the Aug 3 - Aug 14 milestone Jul 31, 2020
@andygrove andygrove self-assigned this Jul 31, 2020
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Aug 3, 2020
@jlowe
Copy link
Member

jlowe commented Aug 3, 2020

Note that when we use compression for shuffle we will likely rely on GpuColaesceBatches to batch-decompress a set of compressed ColumnarBatch. See #487 for details.

@andygrove
Copy link
Contributor Author

After spending time studying the code today, I now understand that when we have GpuCoalesceBatches(GpuShuffleExchangeExec()), what this translates to at execution time is that GpuCoalesceBatches is really wrapping ShuffledBatchRDD which reads the output from the map-side shuffle operation.

The current AQE PR does something very similar because it creates GpuCoalesceBatches(GpuCustomShuffleReader()) which also just wraps ShuffledBatchRDD.

This means we did not lose any optimization after all, so I don't think anything actually needs to happen with this issue.

tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants