Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to drop task or flow return values from memory #7174

Merged
merged 6 commits into from
Oct 18, 2022

Conversation

zanieb
Copy link
Contributor

@zanieb zanieb commented Oct 14, 2022

Adds a cache_result_in_memory option to flows and tasks to drop a result from memory instead of holding it. You can use this while disabling persistence which is the equivalent of saying "I shall not use this result downstream". By default, using it will enable persistence.

Example

See the examples in the documentation

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • This pull request includes tests or only affects documentation.
  • This pull request includes a label categorizing the change e.g. fix, feature, enhancement

@netlify
Copy link

netlify bot commented Oct 14, 2022

Deploy Preview for prefect-orion ready!

Name Link
🔨 Latest commit 518b4e4
🔍 Latest deploy log https://app.netlify.com/sites/prefect-orion/deploys/634dca18915d1900083171a5
😎 Deploy Preview https://deploy-preview-7174--prefect-orion.netlify.app/concepts/results
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@zanieb zanieb added the enhancement An improvement of an existing feature label Oct 14, 2022
@zanieb zanieb marked this pull request as ready for review October 14, 2022 16:07
Copy link
Contributor

@tpdorsey tpdorsey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Minor comments

docs/concepts/results.md Outdated Show resolved Hide resolved
src/prefect/flows.py Outdated Show resolved Hide resolved
docs/concepts/results.md Outdated Show resolved Hide resolved
@zanieb
Copy link
Contributor Author

zanieb commented Oct 17, 2022

I'm thinking that I should toggle persist_result from None to True when this is used so that the default experience of turning it on is better.

docs/concepts/results.md Outdated Show resolved Hide resolved
Copy link
Contributor

@billpalombi billpalombi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both the proposed names and defaults make sense to me.

Copy link
Contributor

@bunchesofdonald bunchesofdonald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it!

@zanieb zanieb merged commit 9df9a14 into main Oct 18, 2022
@zanieb zanieb deleted the results/27-reduce-memory branch October 18, 2022 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants