Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CompressedSerializer for compression of other result serializers #7164

Merged
merged 2 commits into from
Oct 14, 2022

Conversation

zanieb
Copy link
Contributor

@zanieb zanieb commented Oct 13, 2022

Adds CompressedSerializer similar to the one in v1 which lets you use arbitrary (defaults to lzma) compression libraries to compress the bytes returned by an arbitrary serializer.

Documentation preview

Example

@flow(result_serializer="compressed/json")
...

@flow(result_serializer="compressed/pickle")
...

@flow(result_serializer=CompressedSerializer(compressionlib="zlib", serializer="pickle"))
...

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • This pull request includes tests or only affects documentation.
  • This pull request includes a label categorizing the change e.g. fix, feature, enhancement

@netlify
Copy link

netlify bot commented Oct 13, 2022

Deploy Preview for prefect-orion ready!

Name Link
🔨 Latest commit 0a33d95
🔍 Latest deploy log https://app.netlify.com/sites/prefect-orion/deploys/634990a4bb77d0000839c678
😎 Deploy Preview https://deploy-preview-7164--prefect-orion.netlify.app/api-ref/prefect/serializers
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@zanieb zanieb added the enhancement An improvement of an existing feature label Oct 13, 2022
@zanieb zanieb requested review from bunchesofdonald and removed request for cicdw October 13, 2022 22:30
Copy link
Contributor

@tpdorsey tpdorsey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor edit suggestions. Looks good.

src/prefect/serializers.py Outdated Show resolved Hide resolved
src/prefect/serializers.py Outdated Show resolved Hide resolved
src/prefect/serializers.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@chrisguidry chrisguidry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely!


You may configure compression of results using:

- A type name, prefixed with `compressed/` e.g. `"compressed/json"` or `"compressed/pickle"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to do a generalized dispatch implementation but it got really scary :D

Co-authored-by: Terrence Dorsey <terrence@prefect.io>
@peytonrunyan peytonrunyan self-requested a review October 14, 2022 17:02
@zanieb zanieb merged commit 295fd5d into main Oct 14, 2022
@zanieb zanieb deleted the results/26-compression-ser branch October 14, 2022 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants