-
Notifications
You must be signed in to change notification settings - Fork 154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Controlling what gets uploaded to Lambda zip file #35
Comments
Do we think it's ok to include everything by default, and then have user exclude things they don't want uploaded? Or should all inclusion be explicit? |
This is probably related to https://github.com/pulumi/pulumi-service/issues/122 and https://github.com/pulumi/pulumi-service/issues/22. /cc @ellismg |
Related to those, but technically different. There’s two phases:
This is about the second. It may be we decide to offer one mechanism shared across both, but we may not. I’d be inclined to upload everything by default, and let folks remove. But we also should look at Serverless and others for inspiration since they’ve gone through this process already. |
One additional related issue is native modules. When development is done on Mac/Windows, these will not be valid ELF binaries and although they will be uploaded to the Lambda, they will not work. This is a major known issue with Lambda, and we don't exactly make it worse, but we don't help (and we abstract away that it's Lambda, so it's not as easy for us to make the same excuses as Lambda does). There are a few options:
|
i think, sadly, we are looking at option 2 for the short term ... sort of a
semi-known pitfall ... however there are q/a's on the interweb for how to
make the best of this.
i do wonder if we should start to archive (or tag) things for blog posts or
faqs or qas or something ... there's a bunch of times when i read the "real
world knowledge" you are generating and i don't know how to capture.
#PULUMI_FAQ
…On Thu, Oct 26, 2017 at 10:18 AM, Luke Hoban ***@***.***> wrote:
One additional related issue is native modules. When development is done
on Mac/Windows, these will not be valid ELF binaries and although they will
be uploaded to the Lambda, they will not work.
This is a major known issue with Lambda, and we don't exactly make it
worse, but we don't help (and we abstract away that it's Lambda, so it's
not as easy for us to make the same excuses as Lambda does).
There are a few options:
1. If our PPC took only source and ran "npm install" itself - then at
least when using a PPC this would work (by us taking on the burden of being
the CI)
2. For local development, instead of directly uploading the folder, we
could copy it (without node_modules), and then run npm install inside
a Docker container. This would most likely introduce as many or more
problems than it solves, but is probably the only way to get this working
locally.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#35 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AH_5wlS2t6s9rjibfRM6GOs6mZ673maBks5swL7fgaJpZM4Pn_zM>
.
|
@ericrudder We could start a wiki with "known issues" and link to GitHub issues. Later, when we want to release, we can see which issues are still open. |
This has been a huge pain point in the Learning Machine bringup because any change to the source tree invalidates all Lambda resources in the project. |
I second @mmdriley's feedback. This adds about a minute of overhead to each LM publish, even when the lambdas weren't essential and certainly didn't change. Sadly, I have few suggestions on what to do here, because it's JavaScript and could It'd be good to look at Up and Apex to see if they have any clever solution for this. |
I like Luke's suggestion of having an exclude/include file similar to Serverless Framework: https://serverless.com/framework/docs/providers/aws/guide/packaging/ |
So pretty similar to Serverless Framework. I'm pretty convinced that is the right answer here too. However, we have a challenge that there are two steps, both of which will eventually want a
The use cases we're discussing are about (2). I suggest we implement a |
Do we currently do any kind of webpack/browserify step before uploading the code? |
No. Those almost always change semantics of require application-specific knowledge/configuration.
I can imagine that too - but I think the 90% case of the pain here is assets which have no business going to Lambda at all. I think we should fix that. Neither Serverless Framework nor Up have yet addressed the customer-per-lambda case either. Notably - the soltuion to per-lambda customization is probably going to be the same here as for other reasons we might want to customize it: pulumi/pulumi-cloud#168 I think in a model where we allow an explicit |
Given that we're already transforming the user's code, why not have an option to do a webpack? In Azure Functions, there's a preview tool that does this and it worked seamlessly for most customers. It had a big improvement on function size. Agreed about the per-lambda customization case, which we can punt for now. |
Perhaps - depends on whether it's something we could reliably do on the users behalf. Do you have pointers on this? Or suggestions on how we could add something similar to our workflow (which is not quite 1:1 with direct Azure Functions user workflow). |
@lukehoban There's probably a lot more complexity here that I don't entirely understand. :) I assumed we could do something similar to this Serverless Framework plugin: https://www.npmjs.com/package/serverless-webpack. It looks like there's some customization involved, so maybe that's a solution. |
/cc @ellismg There's definitely a deeper topic brewing here, which is whether we "build" code on behalf of the user and, if so, what extensibility mechanisms we offer. As we make progress on the service, I think we're going to have to do increasingly more of this, as @ellismg and I have discussed before. In fact, our current I do like, however, that architecturally these capabilities are layered distinctly. On one hand, the Docker builds bug me because they are the sole instance where we've violated this. On the other hand, I love it, because of its beautiful simplicity.
|
FYI, it looks like Serverless Framework does have a feature for packing Lambdas individually, via the The |
Also /cc'ing @ellismg , since this ties in closely with https://github.com/pulumi/pulumi-service/issues/22 |
An `aws.serverless.Function` can now accept a list of include paths, which will be included inside the Lambda package along with the generated `__index.js` file Also changes the default from '.' to './node_modules', so that the very common case of changes to the root folder do not trigger changes to the code for lambdas. If files from the local folder are needed by the lambda, they can be included in the `includePaths` parameter. Also adds support for providing a pre-created Role instead of just Policies, to support scenarios where users want to pre-create or share a Role across multiple functions. Fixes #35.
With #34 we now upload the entire source folder as part of the Lambda zip. Previously, we uploaded the entire
node_modules
folder.It is in general necessary to have the full source folder to support arbitrary
require('./foo')
in code running "on the inside".However, for many/most Lambdas, they will not use any of the files in the root folder or even the
node_modules
on the inside. So we are carrying these along even though they won't be used.Additionally, in local workflows, the working directory may contain source files or sensitive data that should not be uploaded into the runtime environment.
We could solve this with a configuration setting (possibly in
Pulumi.yaml
) along the lines of https://serverless.com/framework/docs/providers/aws/guide/packaging/.The text was updated successfully, but these errors were encountered: