Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix JSON serialization error when creating flow runs #7385

Merged
merged 4 commits into from
Nov 3, 2022

Conversation

peytonrunyan
Copy link
Contributor

@peytonrunyan peytonrunyan commented Oct 31, 2022

Closes #6962

flow.serialize_parameters is called when the client creates a flow run. This can return invalid JSON in certain cases, such as when serializing dataframes. pd.DataFrame({'col':['1']}) -> {"col": {0: "1"}}.

This resulted in an error when calling FlowRunCreate.dict(json_compatible=True).

This pull request modifies the FlowRunCreate to use orjson.dumps with OPT_NON_STR_KEYS enabled to handle these cases. The code is modified at the child-level because allowing non-string keys has performance implications, so this limits the hit to creating flow runs.

I tried a very simple flow with print statements to check that this is behaving as expected in aggregate (the combination of parameter serialization + flow run serialization), but I'd like a sanity check from @madkinsz to make sure that I'm not missing something.

from prefect import task, flow
import pandas as pd

@task
def task1():
    return 'complete'
    
@flow
def flow1(df):
    print(df.dtypes)
    print(df.head())
    task()
    
df_works = pd.DataFrame({'col':[1]})
flow1(df_works)

df_used_to_error = pd.DataFrame({'col':['1']})
flow1(df_used_to_error)
08:56:01.701 | INFO    | prefect.engine - Created flow run 'masterful-lorikeet' for flow 'flow1'
col    int64
dtype: object
   col
0    1
08:56:01.921 | INFO    | Flow run 'masterful-lorikeet' - Finished in state Completed()
08:56:02.029 | INFO    | prefect.engine - Created flow run 'mysterious-jackdaw' for flow 'flow1'
col    object
dtype: object
  col
0   1
08:56:02.201 | INFO    | Flow run 'mysterious-jackdaw' - Finished in state Completed()

Example

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • This pull request includes tests or only affects documentation.
  • This pull request includes a label categorizing the change e.g. fix, feature, enhancement

@netlify
Copy link

netlify bot commented Oct 31, 2022

Deploy Preview for prefect-orion ready!

Name Link
🔨 Latest commit 4fcadcf
🔍 Latest deploy log https://app.netlify.com/sites/prefect-orion/deploys/635fc6801e2d7c0008cff964
😎 Deploy Preview https://deploy-preview-7385--prefect-orion.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@peytonrunyan peytonrunyan added v2 fix A fix for a bug in an existing feature labels Oct 31, 2022
@anna-geller
Copy link
Contributor

Quick QA:

Version:             2.6.5
API version:         0.8.3
Python version:      3.10.6
Git commit:          9fc2658f
Built:               Thu, Oct 27, 2022 2:24 PM
OS/Arch:             darwin/arm64
Profile:             dev
Server type:         cloud

image

Using this branch, works:
image

@peytonrunyan peytonrunyan marked this pull request as ready for review October 31, 2022 13:43
@zanieb
Copy link
Contributor

zanieb commented Nov 2, 2022

@zangell44 any qualms?

I like that this is scoped to the create action!

@zanieb zanieb merged commit f25f035 into main Nov 3, 2022
@zanieb zanieb deleted the serializer-representation branch November 3, 2022 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix A fix for a bug in an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flow.serialize_parameters can return invalid JSON representation
3 participants