Spike: How to Speed up CDK deployment?

[l0b0]

Look into possibilities for speeding up the CDK deployment, which currently takes quite a long time. Options include:

• Caching pip downloads (find some way to avoid "WARNING: The directory '/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag." when synthesizing)
• Build a Lambda layer with botocore, the biggest package by far
• 'New' synthesize method (didn't speed up things)

Non-options:

• Contrary to AWS recommendations it looks like using the bundled boto3 might not be a good idea.
• I tried using the Dockerfile for both the ECS container image for the batch jobs and for the Lambdas, but it seems they work very differently. For example, I can only pass a single argument as the CMD, and I don't think I can do that and also reuse code which is in the parent directory of the job code.
• I tried the experimental aws_lambda_python.PythonFunction, but found two issues: the runtime is not respected and it generates requirements files which pip doesn't understand (which should've been fixed in the current version of Poetry, 1.1.6).
• See if anything more can be excluded from cdk.out (ncdu cdk.out) - No, the vast majority of the space is taken up by the botocore package now.
• Caching CDK outputs - CDK already does this by default
• Making Docker builds reproducible, by checking that repeated builds of each step results in the same hash. Docker already does this. Also, most of the time-consuming builds are not within Docker.