Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] support barrier mode for mapInPandas/mapInArrow #10344

Closed
wbo4958 opened this issue Jan 31, 2024 · 3 comments · Fixed by #10343 or #10375
Closed

[FEA] support barrier mode for mapInPandas/mapInArrow #10344

wbo4958 opened this issue Jan 31, 2024 · 3 comments · Fixed by #10343 or #10375
Labels
feature request New feature or request

Comments

@wbo4958
Copy link
Collaborator

wbo4958 commented Jan 31, 2024

Is your feature request related to a problem? Please describe.

apache/spark#40520 has introduced barrier mode for mapInPandas/mapInArrow, I hope spark-rapids can support this feature.

How to test?

spark.range(1, 10, 1, 100).mapInPandas(lambda x: x, "id long", True)

to check if all tasks are launched at the same time.
@wbo4958 wbo4958 added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jan 31, 2024
@wbo4958 wbo4958 mentioned this issue Jan 31, 2024
Merged
@sameerz sameerz mentioned this issue Feb 2, 2024
Merged
@wbo4958 wbo4958 mentioned this issue Feb 2, 2024
Merged
@wbo4958 wbo4958 reopened this Feb 2, 2024
@wbo4958
Copy link
Collaborator Author

wbo4958 commented Feb 2, 2024

PR got merged for branch 24.04. Close this issue.

@wbo4958 wbo4958 closed this as completed Feb 2, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Feb 2, 2024
@revans2 revans2 reopened this Feb 2, 2024
@revans2
Copy link
Collaborator

revans2 commented Feb 2, 2024

reopening because it is being reverted again in 24.04 because it does not build on databricks

@wbo4958 wbo4958 mentioned this issue Feb 5, 2024
Merged
@wbo4958
Copy link
Collaborator Author

wbo4958 commented Feb 7, 2024

PR got merged. close it

@wbo4958 wbo4958 closed this as completed Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
3 participants
@wbo4958 @revans2 @mattahrens