Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling cuDF spilling should also enable it on the client. #1391

Closed
ayushdg opened this issue Oct 1, 2024 · 1 comment
Closed

Enabling cuDF spilling should also enable it on the client. #1391

ayushdg opened this issue Oct 1, 2024 · 1 comment

Comments

@ayushdg
Copy link
Member

ayushdg commented Oct 1, 2024

Currently specifying --enable-cudf-spill or enable_cudf_spill=True enables cuDF spilling on the workers, but not the client which leads to errors like cannot create <class 'cudf.core.buffer.spillable_buffer.SpillableBufferOwner'> without a global spill manager

Since spilling is an option that needs to be constantly set on both the client and workers, enabling this when starting up the workers should also set the same option on the client to avoid both environments being out of sync.

cc: @VibhuJawa

@ayushdg ayushdg changed the title Setting cuDF spilling should also enable it on the client. Enabling cuDF spilling should also enable it on the client. Oct 1, 2024
@ayushdg
Copy link
Member Author

ayushdg commented Oct 1, 2024

Actually looks like #1369 covered this for 24.10. It's only 24.8 that doesn't have this behavior.

@ayushdg ayushdg closed this as completed Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant