Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support potential repartitioning for all ExecutionPlans (not just ParquetExec and CSVExec) #7935

Closed
alamb opened this issue Oct 26, 2023 · 0 comments · Fixed by #7936
Closed
Labels
enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented Oct 26, 2023

Is your feature request related to a problem or challenge?

Inspired by PR #7745, the repartition input code as it is now is both a bit messy (needs a #cfg) but also only works for Parqeuet and CSV, where the code is general for any ExecutionPlan.

https://github.com/apache/arrow-datafusion/blob/0911f1523ec7088bae88684ecb9bca94aa553693/datafusion/core/src/physical_optimizer/enforce_distribution.rs#L1270-L1288

Describe the solution you'd like

I would like the code to a trait on ExecutionPlan, which also allows other (user defined) table sources to benefit from this optimization then implement it for Parquet and CSV

Describe alternatives you've considered

No response

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant