Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for using callable objects as tasks #7217

Merged
merged 2 commits into from
Oct 19, 2022
Merged

Add support for using callable objects as tasks #7217

merged 2 commits into from
Oct 19, 2022

Conversation

jmg-duarte
Copy link
Contributor

@jmg-duarte jmg-duarte commented Oct 18, 2022

Fixes #7196

Example

The following example should now work:

from typing import Any

import prefect

class CallableObj:
    def __init__(self) -> None:
        self.x = 10

    def __call__(self, *args: Any, **kwds: Any) -> Any:
        print(self.x)


@prefect.flow
def test_flow():
    t = prefect.Task(fn=CallableObj())
    t()

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • This pull request includes tests or only affects documentation.
  • This pull request includes a label categorizing the change e.g. fix, feature, enhancement

@netlify
Copy link

netlify bot commented Oct 18, 2022

Deploy Preview for prefect-orion ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit 7ee264d
🔍 Latest deploy log https://app.netlify.com/sites/prefect-orion/deploys/634ed419d4d54c00080827ee
😎 Deploy Preview https://deploy-preview-7217--prefect-orion.netlify.app/api-ref/prefect/tasks
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@@ -26,6 +26,8 @@ def to_qualified_name(obj: Any) -> str:
Returns:
str: the qualified name
"""
if not hasattr(obj, "__qualname__"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we update this in tasks/flows.py to call to_qualified_name(type(obj)) instead? to_qualified_name should throw an error for instances because we will not be able to recreate it in from_qualified_name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, I don't think that is a good idea because it will break existing behavior (?)

In [1]: def f():
   ...:     pass
   ...: 

In [2]: type(f).__qualname__
Out[2]: 'function'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh we'd still want to include some if logic, e.g. if not inpect.isfunction(obj): ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, moving the logic to Task.__init__, something like:

if not hasattr(obj, "__qualname__"):
    self.task_key = to_qualified_name(type(obj))
else:
    self.task_key = to_qualified_name(obj)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep!

@zanieb zanieb added the enhancement An improvement of an existing feature label Oct 18, 2022
@zanieb zanieb changed the title Add support for objects to to_qualified_name Add support for using callable objects as tasks Oct 18, 2022
@zanieb zanieb merged commit b775883 into PrefectHQ:main Oct 19, 2022
@jmg-duarte jmg-duarte deleted the fix/7196 branch October 19, 2022 15:02
@zanieb
Copy link
Contributor

zanieb commented Oct 19, 2022

@jmg-duarte The following test fails

    def test_task_run_from_callable_object(self):
        class Foo:
            message = "hello"

            def __call__(self, prefix: str, suffix: str) -> Any:
                return prefix + self.message + suffix

        obj = Foo()
        foo = Task(fn=obj)

        @flow
        def bar():
            return foo("a", suffix="b")

        assert bar() == "ahellob"

E AttributeError: 'Foo' object has no attribute '__name__'

I may have to revert this change. Do you have time to look at a fix?

@jmg-duarte
Copy link
Contributor Author

@madkinsz as a quick fix: adding name=... to the Task call works.
As for an actual fix, I suggest using a similar strategy to the one for the task_key. If that's ok with you, I can submit a fix using that

@zanieb
Copy link
Contributor

zanieb commented Oct 19, 2022

Ah I see I didn't even notice that it was in the __init__ block. Yeah can you remove the name= from your existing test and also include this test that it runs okay? I'm worried we'll inadvertently break something by accessing something that's not present at runtime as well.

@jmg-duarte
Copy link
Contributor Author

also include this test that it runs okay?

Sorry, I don't understand what you mean by this and by extension, which path forward to take: revert or apply the suggestion I gave

@zanieb
Copy link
Contributor

zanieb commented Oct 19, 2022

Sorry! Can you:

  • Apply the suggestion you gave in a new pull request
  • Remove the name=... from the test you added in this pull request so the fix is covered by the test suite
  • Include the test I wrote here to ensure that the task can be used in a flow in creation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tasks do not support callable objects
2 participants