Allow Lambda targets to point to python_aws_lambda_layers and exclude code automatically

[huonw]

Is your feature request related to a problem? Please describe.

In #19123, Pants grows support for python_aws_lambda_layer to package layers for AWS Lambda. These are a bunch of code that's loaded into the Lambda environment, in addition to the core package. When packaging an in-repo layer for use with in-repo functions, typically the function doesn't need to include the code in the layer. Pants could automatically manage this, without needing to specify include_requirements=False and/or dependencies=["!!foo", "!!bar"].

Describe the solution you'd like

Potentially a layers field that takes a list of layer targets. To begin with, this can be just used for excluding the dependencies. This field can be used on both functions and other layers.

(NB. a package built like this is cannot be used in isolation, only with the right layers hooked into the deployed function in AWS.)

For instance, in this example:

• boto3-layer includes the boto3 library and its dependencies, like requests
• func depends on boto3, requests and numpy

The layers field means pants would notice that boto3 (and, preferably, its transitive dependencies like requests, which may require #12733) will already be provided by boto3-layer, and thus only include numpy and foo.py in the func package.

# BUILD
python_requirements(name="reqs") # includes boto3, requests, numpy, etc.

python_aws_lambda_layer(
    name="boto3-layer"
    dependencies=[":reqs#boto3"],
    ...
)

python_aws_lambda_function(
    name="func",
    handler="./foo.py:handler",
    layers=[":boto3-layer"], # NEW FEATURE
    ...
)

# foo.py
import boto3
import requests
import numpy

def handler(...): ...

Describe alternatives you've considered

The current approach of manually spelling out dependencies and exclusions probably works fine for many cases, e.g. a naive "requirements in a layer"/"sources in the function" split is trivial via include_requirements/include_sources.

Additional context
There's a long discussion in #19123 that touches on this.

[huonw]

and, preferably, its transitive dependencies like requests, which may require #12733

Doing this probably also requires pex-tool/pex#2097: e.g. pants would enumerate all the requirements in the parent layer(s), and tell PEX to specifically not include them (against it's better judgement 😅 ).

[huonw]

This would make it easy to have code implicitly use the packages that AWS automatically provides (e.g. boto3), or an external-provided layer (e.g. https://docs.powertools.aws.dev/lambda/python/latest/#lambda-layer) by reifying those layers, so pants can reason about them, but not using them when publishing:

# dependencies provided by the AWS Lambda runtime
python_aws_lambda_layer(
    name="aws-runtime-implicit-layer"
    dependencies=[":reqs#boto3", ...]
)
# dependencies provided by the AWS-published Powertools layer
python_aws_lambda_layer(
    name="aws-powertools-implicit-layer"
    dependencies=[":reqs#aws-lambda-powertools", ...],
    layers=[":aws-runtime-implicit-layer"]
)

python_aws_lambda_layer(
    name="func-deps"
    layers=[":aws-powertools-implicit-layer"], 
    dependencies=["./foo.py"]
)
python_aws_lambda_function(
    name="func",
    dependencies=["./foo.py"]
    layers=[":func-deps"], 
)

Then, to deploy:

1. only func-deps is published as a layer (not aws-powertools-implicit-layer or aws-runtime-implicit-layer)
2. func is published as a function, referencing the AWS-published Powertools layer and the func-deps layer

If this is a useful pattern, Pants can presumably grow automatic management of it (although if pants is managing it, it'll need to have a point of view on version skew between the layers and the runtime environment, e.g. I assume the AWS runtime updates boto3 semi-regularly, and thus may be on a different version to what the user's code specifies).

[huonw]

With pex 2.1.151 (https://github.com/pantsbuild/pex/releases/tag/v2.1.151), which Pants now includes (#20149), there's a flag --exclude that could be used as a building block here: if Pants knows which dependencies are provided by a target environment, they can be --exclude'd.

There's also future plans for --provided (pex-tool/pex#2097 (comment)) too, but I'm not sure it's useful for this, as these lambda artifacts have no PEX initialisation and thus there's little value in --provided, that's able to verify that they actually exist at runtime.