Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] support 4 KiB alignment when reading shuffle buffers #2492

Closed
rongou opened this issue May 24, 2021 · 1 comment
Closed

[FEA] support 4 KiB alignment when reading shuffle buffers #2492

rongou opened this issue May 24, 2021 · 1 comment
Labels
task Work required that improves the product but is not user facing

Comments

@rongou
Copy link
Collaborator

rongou commented May 24, 2021

Is your feature request related to a problem? Please describe.
When GDS spill is enabled, we would like to perform aligned IO, i.e. reads and writes between GPU memory and NVMe drives should be aligned to 4 KiB. However, the current implementation in the shuffle server (specifically in BufferSendState) packs shuffle buffers into the UCX bounce buffer, making it impossible to do aligned reads.

Describe the solution you'd like
An option in the shuffle server to allow 4 KiB aligned reads of shuffle buffers.

Describe alternatives you've considered
An alternative is to unspill whole shuffle buffers back into GPU memory before streaming them to the UCX bounce buffer, but for large shuffle buffers this might cause more spilling, thus less efficient.

Additional context
The GPUDirect Storage best practices guide talks about aligned vs. unaligned IO (https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html).

@abellina @jlowe

@rongou rongou added feature request New feature or request ? - Needs Triage Need team to review and classify labels May 24, 2021
@sameerz sameerz added task Work required that improves the product but is not user facing and removed ? - Needs Triage Need team to review and classify feature request New feature or request labels May 25, 2021
@sameerz
Copy link
Collaborator

sameerz commented May 25, 2021

@rongou can we target this change to the 21.08 release, given that we are in burndown?

@rongou rongou mentioned this issue Jun 4, 2021
11 tasks
@rongou rongou closed this as completed Jan 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
task Work required that improves the product but is not user facing
Projects
None yet
Development

No branches or pull requests

2 participants