Implement layout="zip" for Lambda/GCF, skipping lambdex #19022

huonw · 2023-05-17T11:37:12Z

This fixes #18879 by allowing the python_awslambda and python_google_cloud_function FaaS artefacts to be generated in "simple" format, using the pex3 venv create --layout=flat-zipped functionality recently added in PEX 2.1.135 (https://github.com/pantsbuild/pex/releases/tag/v2.1.135). This format is just: put everything at the top-level, e.g. the zip contains cowsay/__init__.py etc., rather than .deps/cowsay-....whl (plus the dynamic PEX initialisation).

This shifts the dynamic dependency computation/extraction/layout from run-time to build-time, relying on the FaaS environment to be generally consistent. It shouldn't change what actually happens after initialisation. This can:

reduce cold-starts noticeably: for instance, some of our lambdas spend 1s doing PEX/Lambdex start up.
reduce package size somewhat (the PEX .bootstrap/ folder seems to be about 2MB uncompressed, ~1MB compressed).
increase build times.

For instance, for one Python 3.9 Lambda in our codebase:

metric	before	after
init time on cold start	2.3-2.5s	1.3-1.4s (-1s)
compressed size	24.6MB	23.8MB (-0.8MB)
uncompressed size	117.8MB	115.8MB (-2.0MB)
PEX-construction build time	~5s	~5s
PEX-postprocessing build time	0.14s	4.8s

(The PEX-postprocessing time metric is specifically the time to run the Setting up handler (lambdex) or Build python_awslambda (pex3 venv create) process, computed by running pants --keep-sandboxes=always package ... for each layout, and then hyperfine -r3 -w1 path/to/first/__run.sh path/to/second/__run.sh. This doesn't include the time to construct the input PEX, which is the same for both.)

This functionality is driven by adding a new layout field. It defaults to lambdex (retaining the current code paths), but also supports zip, which keys into the functionality above. I've tried to keep the non-lambdex implementation generally separate to the lambdex one, rather than reusing all of the code that happens to be common currently, because it'd make sense to deprecate/remove the lambdex functionality and thus I feel it's best for this new functionality to be mostly a fresh start.

This PR's commits can be reviewed independently. It comes in three phases:

Add the pex_venv.py util rules for running pex3 venv create .... Currently this only supports a limited subset of the functionality there, but can presumably be expanded freely as required. (First commit)
Do some minor refactoring. (Commits labelled "refactor: ...")
Draw the rest of the owl. (The others.)

I think this is an acceptable MVP for this functionality, but there's various bits of follow-up:

deprecate layout="lambdex" (in favour of layout="zip" and/or normal pex_binary) (Deprecate lambdex for building FaaS packages (pending) #19032)
add a warning about files being loaded into these packages, which has been temporarily lost (Warn about files() in PEXes in one place #19027)
adjust documentation
other improvements like Include appropriate complete_platforms for serverless/FaaS environments runtime #18195 and Allow building AWS Lambda Layers #18880
improve performance, e.g. potentially pex3 venv create ... could use the lock file and sources to directly compute the appropriate files, without having to materialise a normal pex first

huonw · 2023-05-17T11:38:21Z

src/python/pants/backend/python/util_rules/pex_venv.py

+
+
+@dataclass(frozen=True)
+class PexVenv:


I suspect this name will be confusing with VenvPex, but I lost inspiration, let me know if you have a better name.

huonw · 2023-05-17T11:42:09Z

src/python/pants/backend/python/util_rules/faas.py

+    pex_request = PexFromTargetsRequest(
+        addresses=[request.address],
+        internal_only=False,


This feels unnecessarily heavyweight, and having to materialise the full non-internal PEX just to throw it away is presumably a major reason why this new approach is slower (since I imagine there's duplicated work between "build a PEX" and "turn that PEX into a venv"), related to the discussion in the PR.

However, this needs to be built for a potentially-foreign platform, and hence, AIUI, internal_only=True and VenvPex don't work.

benjyw · 2023-05-17T16:07:40Z

Hooray! After this I think we should deprecate lambdex and get rid of it. Will review asap.

benjyw · 2023-05-17T23:11:03Z

src/python/pants/backend/awslambda/python/rules.py

-            # The AWS-facing handler function is always lambdex_handler.handler, which is the
-            # wrapper injected by lambdex that manages invocation of the actual handler.
-            handler_log_message="lambdex_handler.handler",
+            # this isn't built or run via lambdex, but the name is arbitrary, so we stick with the


I think this is a good opportunity to get rid of this name, since we're churning anyway. It'll be harder to do later.

But also, are we sure we want this at all? Why not use the handler name the user configured directly?

But also, are we sure we want this at all? Why not use the handler name the user configured directly?

Yes, I think so. The BUILD file target must contain the entry point (at least the file/module part of entry_point="path/to.py:handler"), in order to compute dependencies/build the package, and it's annoying to have to carefully reproduce that elsewhere (handler: 'parent.modules.path.to.handler' in templates/IaC code). Synthesising a fixed module avoids that, and is all info pants already knows.

For instance, in our repo, we use AWS CDK, and have a generic PythonLambda construct where the basic "get the code running" config is just pointing it to the package (e.g. foo.bar/target.zip), because it can just set handler: 'lambdex_handler.handler' globally, not requiring customisation per lambda. I think this is really nice!

(There's obviously often per-lambda config required like any extra env vars or permissions, but CDK/the infra configuration is the source of truth there, it's not just directly duplicated from the BUILD files.)

I think this is a good opportunity to get rid of this name, since we're churning anyway. It'll be harder to do later.

Sure, I'm happy to choose another name, e.g. some options:

lambda_function: https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html suggests AWS configures lambda_function.lambda_handler by default, when creating a Python function using the console

index: using "inline code" in AWS's CDK uses index as the file name (https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.Function.html#handler), but... the handler property always needs to be set

python_awslambda, pants_python_awslambda or __pants_lambda or some other pants-y name

something else?

I think we can also make this customisable in future (including the ability to turn it off), e.g. synthetic_handler_module="my_custom_name" or synthetic_handler_module=None, if required/desired.

I've gone with lambda_function, but more than happy to change. (Thought of another option too: main to be consistent with GCF.)

src/python/pants/backend/python/util_rules/faas.py

…ndler`

…)" This reverts commit 2bc078b.

) Reverts #19022. Per #19032 (comment), we're going to change the approach for moving away from Lambdex to a global setting, rather than the per-target `layout` field added in #19022. That change will simplify things, and involve an extra moving part that will be cherry-picked to 2.17, so I think it's better to start from scratch, and then more carefully cherry-pick the parts of #19022 that are relevant.

This fixes #18879 by allowing the `python_awslambda` and `python_google_cloud_function` FaaS artefacts to be generated in "simple" format, using the `pex3 venv create --layout=flat-zipped` functionality recently added in PEX 2.1.135 (https://github.com/pantsbuild/pex/releases/tag/v2.1.135). This format is just: put everything at the top-level. For instance, the zip contains `cowsay/__init__.py` etc., rather than `.deps/cowsay-....whl`. This avoids the need to do the dynamic PEX initialisation/venv creation. This shifts the dynamic dependency computation/extraction/layout from run-time to build-time, relying on the FaaS environment to be generally consistent. It shouldn't change what actually happens after initialisation. This can: - reduce cold-starts noticeably: for instance, some of our lambdas spend 1s doing PEX/Lambdex start up. - reduce package size somewhat (the PEX `.bootstrap/` folder seems to be about 2MB uncompressed, ~1MB compressed). - increase build times. For instance, for one Python 3.9 Lambda in our codebase: | metric | before | after | |---|---|---| | init time on cold start | 2.3-2.5s | 1.3-1.4s (-1s) | | compressed size | 24.6MB | 23.8MB (-0.8MB) | | uncompressed size | 117.8MB | 115.8MB (-2.0MB) | | PEX-construction build time | ~5s | ~5s | | PEX-postprocessing build time | 0.14s | 4.8s | (The PEX-postprocessing time metric is specifically the time to run the `Setting up handler` (lambdex) or `Build python_awslambda` (`pex3 venv create`) process, computed by running `pants --keep-sandboxes=always package ...` for each layout, and then `hyperfine -r3 -w1 path/to/first/__run.sh path/to/second/__run.sh`. This _doesn't_ include the time to construct the input PEX, which is the same for both.) --- This functionality is driven by adding a new option to the `[lambdex].layout` option added in #19074. In #19074 (targeted for 2.17), it defaults `lambdex` (retaining the current code paths). This PR flips the default to the new option `zip`, which keys into the functionality above. I've tried to keep the non-lambdex implementation generally separate to the lambdex one, rather than reusing all of the code that happens to be common currently, because it'd make sense to deprecate/remove the lambdex functionality and thus I feel it's best for this new functionality to be mostly a fresh start. This PR's commits can be reviewed independently. I _think_ this is an acceptable MVP for this functionality, but there's various bits of follow-up: - add a warning about `files` being loaded into these packages, which has been temporarily lost (#19027) - adjust documentation #19067 - other improvements like #18195 and #18880 - improve performance, e.g. potentially `pex3 venv create ...` could use the lock file and sources to directly compute the appropriate files, without having to materialise a normal pex first This is a re-doing of #19022 with a simpler approach to deprecation, as discussed in #19074 (comment) and #19032 (comment). The phasing will be: | release | supports lambdex? | supports zip? | default layout | deprecation warnings | |---|---|---|---|---| | 2.17 (this PR) | ✅ | ✅ | lambdex | if `layout = "lambdex"` is implicit, tell people to set it: recommend `zip`, but allow `lambdex` if they have to | | 2.18 | ✅ | ✅ | zip | if `layout = "lambdex"` is set at all, tell people to remove it and switch to `zip` | | 2.19 | ❌ | ✅ | zip | none, migration over (or maybe just about removing the `[lambdex]` section entirely) |

…d#19076) This fixes pantsbuild#18879 by allowing the `python_awslambda` and `python_google_cloud_function` FaaS artefacts to be generated in "simple" format, using the `pex3 venv create --layout=flat-zipped` functionality recently added in PEX 2.1.135 (https://github.com/pantsbuild/pex/releases/tag/v2.1.135). This format is just: put everything at the top-level. For instance, the zip contains `cowsay/__init__.py` etc., rather than `.deps/cowsay-....whl`. This avoids the need to do the dynamic PEX initialisation/venv creation. This shifts the dynamic dependency computation/extraction/layout from run-time to build-time, relying on the FaaS environment to be generally consistent. It shouldn't change what actually happens after initialisation. This can: - reduce cold-starts noticeably: for instance, some of our lambdas spend 1s doing PEX/Lambdex start up. - reduce package size somewhat (the PEX `.bootstrap/` folder seems to be about 2MB uncompressed, ~1MB compressed). - increase build times. For instance, for one Python 3.9 Lambda in our codebase: | metric | before | after | |---|---|---| | init time on cold start | 2.3-2.5s | 1.3-1.4s (-1s) | | compressed size | 24.6MB | 23.8MB (-0.8MB) | | uncompressed size | 117.8MB | 115.8MB (-2.0MB) | | PEX-construction build time | ~5s | ~5s | | PEX-postprocessing build time | 0.14s | 4.8s | (The PEX-postprocessing time metric is specifically the time to run the `Setting up handler` (lambdex) or `Build python_awslambda` (`pex3 venv create`) process, computed by running `pants --keep-sandboxes=always package ...` for each layout, and then `hyperfine -r3 -w1 path/to/first/__run.sh path/to/second/__run.sh`. This _doesn't_ include the time to construct the input PEX, which is the same for both.) --- This functionality is driven by adding a new option to the `[lambdex].layout` option added in pantsbuild#19074. In pantsbuild#19074 (targeted for 2.17), it defaults `lambdex` (retaining the current code paths). This PR flips the default to the new option `zip`, which keys into the functionality above. I've tried to keep the non-lambdex implementation generally separate to the lambdex one, rather than reusing all of the code that happens to be common currently, because it'd make sense to deprecate/remove the lambdex functionality and thus I feel it's best for this new functionality to be mostly a fresh start. This PR's commits can be reviewed independently. I _think_ this is an acceptable MVP for this functionality, but there's various bits of follow-up: - add a warning about `files` being loaded into these packages, which has been temporarily lost (pantsbuild#19027) - adjust documentation pantsbuild#19067 - other improvements like pantsbuild#18195 and pantsbuild#18880 - improve performance, e.g. potentially `pex3 venv create ...` could use the lock file and sources to directly compute the appropriate files, without having to materialise a normal pex first This is a re-doing of pantsbuild#19022 with a simpler approach to deprecation, as discussed in pantsbuild#19074 (comment) and pantsbuild#19032 (comment). The phasing will be: | release | supports lambdex? | supports zip? | default layout | deprecation warnings | |---|---|---|---|---| | 2.17 (this PR) | ✅ | ✅ | lambdex | if `layout = "lambdex"` is implicit, tell people to set it: recommend `zip`, but allow `lambdex` if they have to | | 2.18 | ✅ | ✅ | zip | if `layout = "lambdex"` is set at all, tell people to remove it and switch to `zip` | | 2.19 | ❌ | ✅ | zip | none, migration over (or maybe just about removing the `[lambdex]` section entirely) |

…ck of #19076) (#19120) This fixes #18879 by allowing the `python_awslambda` and `python_google_cloud_function` FaaS artefacts to be generated in "simple" format, using the `pex3 venv create --layout=flat-zipped` functionality recently added in PEX 2.1.135 (https://github.com/pantsbuild/pex/releases/tag/v2.1.135). This format is just: put everything at the top-level. For instance, the zip contains `cowsay/__init__.py` etc., rather than `.deps/cowsay-....whl`. This avoids the need to do the dynamic PEX initialisation/venv creation. This shifts the dynamic dependency computation/extraction/layout from run-time to build-time, relying on the FaaS environment to be generally consistent. It shouldn't change what actually happens after initialisation. This can: - reduce cold-starts noticeably: for instance, some of our lambdas spend 1s doing PEX/Lambdex start up. - reduce package size somewhat (the PEX `.bootstrap/` folder seems to be about 2MB uncompressed, ~1MB compressed). - increase build times. For instance, for one Python 3.9 Lambda in our codebase: | metric | before | after | |---|---|---| | init time on cold start | 2.3-2.5s | 1.3-1.4s (-1s) | | compressed size | 24.6MB | 23.8MB (-0.8MB) | | uncompressed size | 117.8MB | 115.8MB (-2.0MB) | | PEX-construction build time | ~5s | ~5s | | PEX-postprocessing build time | 0.14s | 4.8s | (The PEX-postprocessing time metric is specifically the time to run the `Setting up handler` (lambdex) or `Build python_awslambda` (`pex3 venv create`) process, computed by running `pants --keep-sandboxes=always package ...` for each layout, and then `hyperfine -r3 -w1 path/to/first/__run.sh path/to/second/__run.sh`. This _doesn't_ include the time to construct the input PEX, which is the same for both.) --- This functionality is driven by adding a new option to the `[lambdex].layout` option added in #19074. In #19074 (targeted for 2.17), it defaults `lambdex` (retaining the current code paths). This PR flips the default to the new option `zip`, which keys into the functionality above. I've tried to keep the non-lambdex implementation generally separate to the lambdex one, rather than reusing all of the code that happens to be common currently, because it'd make sense to deprecate/remove the lambdex functionality and thus I feel it's best for this new functionality to be mostly a fresh start. This PR's commits can be reviewed independently. I _think_ this is an acceptable MVP for this functionality, but there's various bits of follow-up: - add a warning about `files` being loaded into these packages, which has been temporarily lost (#19027) - adjust documentation #19067 - other improvements like #18195 and #18880 - improve performance, e.g. potentially `pex3 venv create ...` could use the lock file and sources to directly compute the appropriate files, without having to materialise a normal pex first This is a re-doing of #19022 with a simpler approach to deprecation, as discussed in #19074 (comment) and #19032 (comment). The phasing will be: | release | supports lambdex? | supports zip? | default layout | deprecation warnings | |---|---|---|---|---| | 2.17 (this PR) | ✅ | ✅ | lambdex | if `layout = "lambdex"` is implicit, tell people to set it: recommend `zip`, but allow `lambdex` if they have to | | 2.18 | ✅ | ✅ | zip | if `layout = "lambdex"` is set at all, tell people to remove it and switch to `zip` | | 2.19 | ❌ | ✅ | zip | none, migration over (or maybe just about removing the `[lambdex]` section entirely) |

There's several places that check whether a PEX transitively depends on any `files()` targets, because those are (currently) not included in the PEX. This moves the warning into the "pex from targets" builder, to not have to reimplement it everywhere. This is follow up to #19022, and restores the warning for `layout="zip"`, since I didn't implement it there.

…19148) There's several places that check whether a PEX transitively depends on any `files()` targets, because those are (currently) not included in the PEX. This moves the warning into the "pex from targets" builder, to not have to reimplement it everywhere. This is follow up to #19022, and restores the warning for `layout="zip"`, since I didn't implement it there. Co-authored-by: Huon Wilson <huon@exoflare.io>

huonw requested a review from benjyw May 17, 2023 11:37

huonw requested review from kaos and thejcannon as code owners May 17, 2023 11:37

huonw commented May 17, 2023

View reviewed changes

huonw added the category:user api change label May 17, 2023

huonw force-pushed the feature/18879-lambda-layout branch from 2c191e3 to 8a19f67 Compare May 17, 2023 11:47

huonw added 7 commits May 18, 2023 07:38

Add rule/targets for 'pex3 venv create' invocations

59ea698

refactor: track module/func separately in ResolvedPythonFaaSHandler

6ff0d48

refactor: pull out logic for major, minor -> interpreter platform

7a0d5c2

New target types

012cd5b

Implement 'venv create'-calling builder rule

4193be5

Allow Lambdas and GCF to build using new rules

0346958

synthesise a handler module

8087903

huonw force-pushed the feature/18879-lambda-layout branch from 8a19f67 to 8087903 Compare May 17, 2023 21:38

huonw mentioned this pull request May 17, 2023

Warn about files() in PEXes in one place #19027

Merged

benjyw approved these changes May 17, 2023

View reviewed changes

huonw mentioned this pull request May 18, 2023

Deprecate lambdex for building FaaS packages (pending) #19032

Closed

huonw added 3 commits May 19, 2023 07:51

Review: docs: artefact -> artifact, recommend layout=zip

82399f7

Review: reexport handler as lambda_function rather than `lambdex_ha…

ec3f56e

…ndler`

Review: document reexporting behaviour

bde2d5a

huonw merged commit 2bc078b into pantsbuild:main May 20, 2023

huonw deleted the feature/18879-lambda-layout branch May 20, 2023 02:07

huonw mentioned this pull request May 20, 2023

Update FaaS documentation for new layout="zip" #19067

Closed

huonw added a commit that referenced this pull request May 20, 2023

Revert "Implement layout="zip" for Lambda/GCF, skipping lambdex (#19022…

813b88d

…)" This reverts commit 2bc078b.

huonw mentioned this pull request May 21, 2023

Add [lambdex].layout to opt-in to soon-to-be-deprecated FaaS behaviour #19074

Closed

huonw mentioned this pull request May 21, 2023

Implement layout="zip" for Lambda/GCF, deprecating lambdex #19076

Merged

WorkerPants mentioned this pull request May 23, 2023

Implement layout="zip" for Lambda/GCF, deprecating lambdex (Cherry-pick of #19076) #19119

Closed

huonw mentioned this pull request May 23, 2023

Implement layout="zip" for Lambda/GCF, deprecating lambdex (Cherry-pick of #19076) #19120

Merged

WorkerPants mentioned this pull request May 25, 2023

Warn about files() in PEXes in one place (Cherry-pick of #19027) #19148

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement layout="zip" for Lambda/GCF, skipping lambdex #19022

Implement layout="zip" for Lambda/GCF, skipping lambdex #19022

huonw commented May 17, 2023 •

edited

Loading

huonw May 17, 2023

huonw May 17, 2023

benjyw commented May 17, 2023

benjyw May 17, 2023

benjyw May 17, 2023

huonw May 17, 2023 •

edited

Loading

huonw May 18, 2023



		@dataclass(frozen=True)
		class PexVenv:

Implement layout="zip" for Lambda/GCF, skipping lambdex #19022

Implement layout="zip" for Lambda/GCF, skipping lambdex #19022

Conversation

huonw commented May 17, 2023 • edited Loading

huonw May 17, 2023

Choose a reason for hiding this comment

huonw May 17, 2023

Choose a reason for hiding this comment

benjyw commented May 17, 2023

benjyw May 17, 2023

Choose a reason for hiding this comment

benjyw May 17, 2023

Choose a reason for hiding this comment

huonw May 17, 2023 • edited Loading

Choose a reason for hiding this comment

huonw May 18, 2023

Choose a reason for hiding this comment

huonw commented May 17, 2023 •

edited

Loading

huonw May 17, 2023 •

edited

Loading