Best Practices for DataTree with Multiscale Data

[allen-adastra]

Mostly pinging @TomNicholas, I see DataTree is making it into the public API; exciting!

I was hoping to get your thoughts on best practices, with a concrete fusion example.

Let's consider magnetic data. We have that at three resolutions:

1. Order 10Hz from pre-shot pulse-design
2. Order 1kHz from real-time equilibrium reconstruction
3. Order 10kHz for the raw magnetic data

We also have many shots of data. Shots tend to be between 0.5 seconds and 3 seconds, which isn't a huge range, but a large enough range that we probably shouldn't just nan pad the ends and have "shot" just be a dimension along an array.

What kind of structure would you recommend with DataTree? Should we just have one node for each "shot"? Is there some built-in functionality to "transpose" a variable? (i.e. go from multiple arrays of "signalX" to a single array of "signalX" with a new dimension and nan padding to handle different lengths).

[TomNicholas]

Order 10Hz from pre-shot pulse-design
Order 1kHz from real-time equilibrium reconstruction
Order 10kHz for the raw magnetic data

It's very satisfying to me to see people taking these same data management concepts and applying them to fusion 😊

probably shouldn't just nan pad the ends and have "shot" just be a dimension along an array.

Yeah that's hopefully a pre-datatree anti-pattern now.

What kind of structure would you recommend with DataTree?

Instead of answering directly yet I'm going to use you as a guinea pig and point you to some upcoming documentation I wrote - PR #9501, the latest build is here. I'm hoping reading that helps answers your question - if it doesn't then please say so.

Is there some built-in functionality to "transpose" a variable? (i.e. go from multiple arrays of "signalX" to a single array of "signalX" with a new dimension and nan padding to handle different lengths).

Nothing dedicated like that exists for datatree yet, but it could exist. In fact that sounds very similar to calling xr.concat with join='outer' & fill_value=np.Nan? (Also this is an example of #9349)

[allen-adastra]

could

True, we can do this. Something I've noticed is that I can't have time coordinates be:

time (shot, time)

for each shot of xr.Dataset.

However, if I instead have time be a data variable and rename the time dimension to time_slice with index coordinates that is just np.arange, then I can do the concatenation. Wondering if this is the best practice?

[allen-adastra]

Docs look good, I'll try to get around to playing around with it; when is the release that contains xr.DataTree going to be out? (I can ofc install HEAD of main, but for my more productiony stuff would be good to have a proper release).