Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom rdds: initial writers module #31

Merged
merged 21 commits into from
Oct 1, 2014

Conversation

industrial-sloth
Copy link
Contributor

Biggest change here is the addition of a new "writers" module, to abstract over writing to various filesystems (currently the local fs and S3) in the same way that readers attempts to abstract away the filesystem being read from. Also includes a number of bug fixes.
Smaller changes:

  • set kryo buffer size to 1GB by default in thunder ec2 script
  • add option in Images exportAsPngs to first collect the pngs to the driver before writing. Should make it easier to get useful output onto a local filesystem.
  • add FileNotFound error in readers for additional abstraction of error cases, currently implemented only for local reader (boto reader is a todo item)
  • fix IndexError in block size calc when requested block size is larger than data set
  • change Images saveAsBinarySeries filename output so that default sorting will group together all files on matching z plane
  • fix reading of conf.json file in SeriesLoader fromBinary()

freeman-lab added a commit that referenced this pull request Oct 1, 2014
Custom rdds: initial writers module
@freeman-lab freeman-lab merged commit 64a10b2 into thunder-project:custom-rdds Oct 1, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants