Options for data transfer between gs buckets and OSN

[norlandrhagen]

We should survey options for data transfer between tmp buckets and long-er term storage.

• fsspec
• object-store
• skyplane 🚧 @jbusecke see if we can establish contact with those folks.
• s5cmd @norlandrhagen
• s5cmd github action
• rclone / rsync @jbusecke
• arraylake 🚧 @jbusecke will reach out to the team there.

[jbusecke]

This would be a great topic for a focused hack/meeting. @leap-stc/data-and-compute who would be interested in joining?

I can make time for this Mon/Tue (Aug 12/13) if that works for others.

[norlandrhagen]

Aug 12th or 13th work for me.

[andersy005]

This would be a great topic for a focused hack/meeting. @leap-stc/data-and-compute who would be interested in joining?

I can make time for this Mon/Tue (Aug 12/13) if that works for others.

count me in... i;m also available on these days ☝🏽

[norlandrhagen]

https://www.when2meet.com/?25865623-leBGV

cc @jbusecke @andersy005 @SammyAgrawal

[jbusecke]

We decided to split this work up (see above), and then reconvene with some intermediate results.

Intermediate goal: Try to write a drop in replacement for the Copy Stage

Lets collect things in this repo

Suggestions for a meeting date?