Include appropriate complete_platforms for serverless/FaaS environments runtime

[huonw]

Is your feature request related to a problem? Please describe.

Currently building Serverless artefacts like AWS Lambda or GCF supports both a quick-and-easy runtime setting, and a complete_platforms. Generally setting complete_platforms is required to get the right wheels, and setting runtime is not useful/misleading/causes problems (e.g. #15296, #18001, and a continual stream of questions in Slack).

Describe the solution you'd like

I think the environments for a given runtime are generally pretty stable, so pants itself could potentially include an appropriate complete platform JSON for each runtime setting (so 3 for AWS Lambda, and 4 for GCF). That is, something like python_awslambda(..., runtime="python39") would be translated into calling pex with a pants-provided complete platform JSON, rather than just a simple platform identifier.

Having pants provide a complete platform almost certainly won't be worse than not using a complete platform at all, and would hopefully mean fewer users have problems. Any users who still have problems can provide their own complete platform.

Downside: if the complete platform pants provides is changed (e.g. pants 2.35 has a new JSON than pants 2.34), built artefacts may change after a user upgrades to 2.35, due to differing wheel selection.

Describe alternatives you've considered

Just not doing this, because the downsides/risks are too large?

Additional context

The complete platforms can be generated by running PEX in each of the relevant environments. For instance, we're deploying AWS Lambdas using a complete platform generated by:

import subprocess

def lambda_handler(event, context):
    subprocess.run(
        """
        pip install --target=/tmp/subdir pex
        PYTHONPATH=/tmp/subdir /tmp/subdir/bin/pex3 interpreter inspect --markers --tags
        """,
        shell=True
    )
    return {
        'statusCode': 200,
        'body': "{}",
    }

I think this would "just" mean switching the following bit of code to select an appropriate pants-provided JSON file, rather than just constructing the platform string:

pants/src/python/pants/backend/awslambda/python/rules.py

Lines 101 to 114 in 4b2bc11

	# We hardcode the platform value to the appropriate one for each AWS Lambda runtime.
	# (Running the "hello world" lambda in the example code will report the platform, and can be
	# used to verify correctness of these platform strings.)
	pex_platforms = []
	interpreter_version = field_set.runtime.to_interpreter_version()
	if interpreter_version:
	py_major, py_minor = interpreter_version
	platform_str = f"linux_x86_64-cp-{py_major}{py_minor}-cp{py_major}{py_minor}"
	# set pymalloc ABI flag - this was removed in python 3.8 https://bugs.python.org/issue36707
	if py_major <= 3 and py_minor < 8:
	platform_str += "m"
	if (py_major, py_minor) == (2, 7):
	platform_str += "u"
	pex_platforms.append(platform_str)

[huonw]

This was done for AWS Lambda in #19253, but not GCF.

[huonw]

Discussed in https://pantsbuild.slack.com/archives/C0D7TNJHL/p1686082038188639 (aka https://chat.pantsbuild.org/t/12319378/both-aws-lambda-and-google-cloud-functions-allows-using-a-zi#b8423452-b0f9-44c1-a6e0-f5c18eb32c27 https://chat.pantsbuild.org/t/12320824/does-anyone-happen-to-have-knowledge-of-gcf-for-the-question#9e9eb90b-e6b3-4b0b-ae88-d0090a65fcec )

[huonw]

This was done for AWS by running an appropriate pex3 ... command (see #19253) in the docker images AWS provides. It seems like GCF doesn't provide such images, so it wasn't easy to do that too.

The current thinking is run the pex3 ... command in the real environment, e.g. deploy a function that calls subprocess.run. I'm personally unsure how to do this yet (never used GCP yet!), so haven't invested the time.

[ryaminal]

decided to run this on the 4 python versions that are available in GCF v2(based on top of cloud run). not certain if it's different for v1, but i can get those also.
gcf_v2_38
gcf_v2_39
gcf_v2_310
gcf_V2_311

no idea if all 4 were needed but... yeah

[huonw]

That's great @ryaminal! More is better! That's great. I'm on leave for a few days so I can't offer much assistance just now, but if you were make a PR adding them to the repo, that'd be cool.

For v1 vs v2, I have no idea. Do you have a sense for how often people use v1?