Storage documents not working as expected from Deployment.build_from_flow followed by Deployment.apply (Prefect2.1.0)

[MiniEggz]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Hi, I've found a little bug with deployments and storage blocks/documents.

When running code very similar to the example @anna-geller gave here, but using an Azure container instead of an S3 bucket, the storage was not a document but just a pointer to the azure storage block in prefect cloud. I managed to find a way to get this working again, but I'm not sure whether it could cause problems elsewhere.

I found the bug by creating the deployment on my local machine, spinning up an agent on an azure vm, and then running the deployment from prefect cloud. The error message just showed that the file was not found (as it was looking for an absolute path on my machine).

The solution I came up with was adding this into line 386 of prefect.deployments.py:

        else:
            # create storage block document from deployments storage block
            self.storage = self.storage.copy(
                exclude={"_block_document_id", "_block_document_name", "_is_anonymous"}
            )
            file_count = await self.storage.put_directory(ignore_file=".prefectignore")

This works for my use case, but I'm not sure if it messes with things elsewhere.

Thanks :)

Reproduction

from flows.healthcheck import healthcheck
from prefect.deployments import Deployment
from prefect.filesystems import Azure

deployment = Deployment.build_from_flow(
    flow=healthcheck,
    name="pythonics3",
    description="hi!",
    version="1",
    work_queue_name="dev",
    tags=["dev"],
    storage=Azure.load("azure-storage-block"),
    infra_overrides=dict(env={"PREFECT_LOGGING_LEVEL": "DEBUG"}),
		output="your.yaml"
)
if __name__ == "__main__":
    deployment.apply()

Error

Versions

2.1.0

Additional context

No response

[anna-geller]

Thanks for opening the issue and describing the problem so nicely.

I couldn't reproduce any issues following the same deployment configuration as the one you shared and I didn't have to change anything on the block document side.

Could you perhaps try the same with CLI without making any code changes to Prefect and see if this fixes the issue for you?
Here is a simple example:

prefect deployment build healthcheck.py:healthcheck --name az -q prod -t project -o az.yaml -sb azure/prod
prefect deployment apply az.yaml
prefect deployment run healthcheck/az

[MiniEggz]

@anna-geller thanks for the reply. I tried creating the flow as suggested above using the CLI and everything's all working there :)

I then went back to create a really simple hello world python flow and deployment (I have also upgraded to v2.1.1 now):

test_flow.py

from prefect import flow


@flow
def hello_world():
    print("Hello World!")

test_dp.py

from prefect import flow
from prefect.deployments import Deployment
from prefect.filesystems import Azure

from test_flow import hello_world

storage_block = Azure.load("azure-storage-block")

deployment = Deployment.build_from_flow(
    name="az",
    flow=hello_world,
    flow_name="hello-world",
    work_queue_name="dev",
    tags=["dev"],
    description="Some description.",
    storage=storage_block,
    output="az.yaml",
)

if __name__ == "__main__":
    deployment.apply()

The deployment went up to prefect cloud just fine. I then ran an agent locally and an error with the path (looking for an absolute path on my machine in the azure container).

After having a look at the prefect cloud, I noticed that on line 479 of deployments.py, if we changed deployment.entrypoint = f"{Path(flow_file).absolute()}:{flow.fn.__name__}" to deployment.entrypoint = f"{Path(flow_file).name}:{flow.fn.__name__}" it fixed the issue. Hoping that works as a fix that doesn't break anything else? :)

[ross-kortx]

Same issue here. Allowing an 'entrypoint' argument to override this behavior would help as well.

YAML with the CLI

entrypoint: hub_prefect/flows/xandr/validate_daily/flow.py:main

YAML with the Python

entrypoint: /Users/rossteach/Documents/code/git/hub-prefect/hub_prefect/flows/xandr/validate_daily/flow.py:main

[anna-geller]

Thanks all for reporting. I couldn't reproduce but I passed the issue along to other engineers, so it might be that someone will have a look next week. Until then, I'd recommend using the CLI, as it remains the primary and recommended way of creating deployments.