WIP: Beam executor without writing to temporary arrays #36
Conversation
This isn't ready for a full review yet since it doesn't have any tests -- I haven't even run the code.
Codecov Report
@@ Coverage Diff @@
## master #36 +/- ##
==========================================
- Coverage 93.39% 84.74% -8.66%
==========================================
Files 7 7
Lines 318 367 +49
Branches 65 76 +11
==========================================
+ Hits 297 311 +14
- Misses 10 44 +34
- Partials 11 12 +1
Continue to review full report at Codecov.
|
I originally thought that it made the most sense to think of rechunking as a chain of chunked array copies. After going through the exercise of trying to write an Executor that doesn't use temporary arrays (pangeo-data#36), I realize now that this was overly generic. *Sometimes* it might make sense to implement a rechunking this way, but Rechunker's algorithm adds more structure than that. There is always a "push/pull" structure that splits read chunks into intermediate chunks, and then combines intermediate chunks into write chunks. This structure wasn't evident from current data model and I needed it for my executor without temporaries, so this PR changes CopySpec to directly keep track of "read/intermediate/write" steps. It's still convenient sometimes to use the "chain of copies" representation, so I've added the utility function `split_into_direct_copies()` for converting to this representation in `rechunker.executors.util`.
|
I'm going to close this. We implemented a variant of this executor inside xarray-beam (but on whole xarray datasets instead of numpy arrays): That would be a good starting point if anyone wants to revisit this. In practice, Beam does seem to typically write out intermediate chunks to temporary storage, but it's still nice to have the whole thing in via Beam transforms, especially for composing into larger pipelines (e.g., |
This isn't ready for a full review yet since it doesn't have any tests -- I
haven't even run the code.