[WIP] Pack/unpack cuDF frames during serialization #4803
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jakirkham
added
2 - In Progress
|
Can one of the admins verify this patch? |
jakirkham
force-pushed
the
agg_frames
branch
4 times, most recently
from
f35dbc5
to
2b40691
Compare
Dask typically does this. So may overwrite the value we have here. However just make sure we have set this value correctly. It makes it a little easier to play with the serialization and deserialization functions here. Plus if Dask doesn't set these for some reason, we have handled it ourselves.
In the `"cuda"` case, we really want to make sure Dask is not trying to compress our data. This shouldn't be happening anyways. However this provides some protection against it. In the `"dask"` case, allow compression since this is data on host.
To cutdown on the number of frames that need to be transmitted over the wire or transferred to/from host, pack all `Buffer`s into one `DeviceBuffer` using CuPy to facilitate. In deserialization, use CuPy to unpack them into a series of separate `DeviceBuffer` allocations. Since cuDF already makes sure CuPy uses RMM for allocations, we need not worry about this when creating `ndarray`s. However it is worth noting we pay for a copy and a larger allocation both during serialization and deserialization. It would be nice to avoid that, but it probably requires some C++/CUDA code to do. So have held off on it for now.
|
Looks like Dask is doing something funky. The gist is it appears to be trying to call |
Codecov Report
@@ Coverage Diff @@
## branch-0.14 #4803 +/- ##
===============================================
- Coverage 88.42% 88.34% -0.09%
===============================================
Files 51 51
Lines 9734 9743 +9
===============================================
Hits 8607 8607
- Misses 1127 1136 +9
Continue to review full report at Codecov.
|
|
Closing as this is more thoroughly addressed by PR ( #5025 ). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to issue ( #3793 ), issue ( https://github.com/rapidsai/rmm/issues/318 ), issue ( rapidsai/dask-cuda#250 )
To cutdown on the number of frames that need to be transmitted over the wire or transferred to/from host, pack all
Buffers into oneDeviceBufferusing CuPy to facilitate. In deserialization, use CuPy to unpack them into a series of separateDeviceBufferallocations.Since cuDF already makes sure CuPy uses RMM for allocations, we need not worry about this when creating
ndarrays. However it is worth noting we pay for a copy and a larger allocation both during serialization and deserialization. It would be nice to avoid that, but it probably requires some C++/CUDA code to do. So have held off on it for now.cc @quasiben