Remote changes to Lambda code do not trigger update

[austinbutler]

This is with a TypeScript project using only aws resources, no awsx ones. Pulumi v1.13.0, @pulumi/aws@1.27.0.

Steps:

1. Make code changes to a Lambda function in the AWS console
2. Run pulumi up -r

At this point in the "details" it will show that there are differences with the Lambda function, such as lastModified, sourceCodeHash, and sourceCodeSize. Yet the summary says 188 unchanged and if I proceed with the up the remote Lambda changes are not reverted.

For example during the up it shows sourceCodeSize: 2762 => 2764. The state at that time (before proceeding) indeed shows 2762. Proceeding, then checking the state again, the state has updated to 2764, but the remote code is unchanged even though it doesn't match the local code.

During imports of existing Lambdas I recall having to comment out the code key for the Lambda in Pulumi otherwise it would give the import will fail due to it not matching error. I'm guessing that might be related? At the time I figured it was some sort of import limitation.

The only way to get the remote code to update is to update the local code. At that point a change is detected and the local code is successfully pushed to remote.

[mikhailshilkov]

I confirmed this is still the case today. Repro:

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

const role = new aws.iam.Role("mylambda-role", {
    assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "lambda.amazonaws.com" }),
});

const policy = new aws.iam.RolePolicy("mylambda-policy", {
    role,
    policy: pulumi.output({
        Version: "2012-10-17",
        Statement: [{
            Action: ["logs:*", "cloudwatch:*"],
            Resource: "*",
            Effect: "Allow",
        }],
    }),
});

const lambda = new aws.lambda.Function("mylambda", {
    runtime: "nodejs18.x",
    code: new pulumi.asset.FileArchive("./app"),
    timeout: 300,
    handler: "bla.js",
    role: role.arn,
}, { dependsOn: [policy] });

[t0yv0]

I had a chance to revisit this problem today, unfortunately this dangerous behavior appears to be something we are inheriting from upstream TF provider, although I could not find an open issue about it in the 30 or so issues pertaining to source_code_hash.

Consider this program:

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "iam_for_lambda" {
  name               = "iam_for_lambda_anton"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

data "archive_file" "lambda" {
  type        = "zip"
  source_file = "lambda.js"
  output_path = "lambda_function_payload.zip"
}

resource "aws_lambda_function" "test_lambda" {
  filename      = "lambda_function_payload.zip"
  function_name = "lambda_function_name"
  role          = aws_iam_role.iam_for_lambda.arn
  handler       = "index.test"

  # source_code_hash = data.archive_file.lambda.output_base64sha256

  runtime = "nodejs18.x"

  environment {
    variables = {
      foo = "bar"
    }
  }
}

If you do not specify source_code_hash, then TF computes it, but it critically relies on it to detect drift. With a computed source_code_hash it does not detect drift, that is:

1. terraform apply
2. edit code in cloud
3. terraform refresh
4. terraform apply
5. observe no changes, program does not match cloud

It appears that prominent examples in TF do recommend to populate source_code_hash in your program. If you do that, you get more reasonable behavior:

1. terraform apply
2. edit code in cloud
3. terraform refresh
4. terraform apply
5. diff shown, cloud updated to match local source

You can get that same reasonable behavior in Pulumi if you populate sourceCodeHash, which is the current recommended workaround for this issue:

import * as archive from "@pulumi/archive";
import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

const role = new aws.iam.Role("mylambda-role", {
    assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "lambda.amazonaws.com" }),
});

const policy = new aws.iam.RolePolicy("mylambda-policy", {
    role,
    policy: pulumi.output({
        Version: "2012-10-17",
        Statement: [{
            Action: ["logs:*", "cloudwatch:*"],
            Resource: "*",
            Effect: "Allow",
        }],
    }),
});

const lambdaArchive = archive.getFile({
    type: "zip",
    sourceFile: "lambda.js",
    outputPath: "lambda_function_payload.zip"
});

const lambdaFunc = new aws.lambda.Function("mylambda", {
    runtime: "nodejs18.x",
    code: new pulumi.asset.FileArchive("./app"),
    timeout: 300,
    sourceCodeHash: lambdaArchive.then(lambda => lambda.outputBase64sha256),
    handler: "bla.js",
    role: role.arn,
}, { dependsOn: [policy] });


export const lambdaARN = lambdaFunc.arn;

[t0yv0]

@corymhall @flostadler brings up we could try auto-computing sourceCodeHash from FileArchive if unspecified, to do better than upstream here; at Pulumi level. This does not immediately appear to have major downsides, so worth trying out.

[corymhall]

There were some recent changes to upstream that changed the behavior a little. Now there are two fields that track the source code hash:

• source_code_hash (input): This is used to trigger updates and is really no longer a computed value
• code_sha256 (output): This is used to store the CodeSha256 value that is part of the function configuration and is computed by the lambda service.

Previously the difficulty was that if you provided the source_code_hash you had to make sure it matched what lambda calculated, otherwise it would create a permanent diff. Now that these are separate fields you no longer need to worry about it. In Pulumi you also do not need to provide sourceCodeHash because pulumi will trigger an update if the code asset changes. You can also provide sourceCodeHash if you want to use it to calculate other scenarios that should cause an update, but it is not needed.

At this point I don't see a path forward since upstream decided to make this change it doesn't seem possible to trigger an update if source_code_hash is different from code_sha256.

[t0yv0]

Checking up on this @corymhall it looks like the change you mention stays that sourceCodeHash is deprecated but they did not mark it as such in the schema; as a result our code is not marking it deprecated either. Should we consider fixing this for the docs?

[corymhall]

it looks like they only deprecated it for the data source not the resource.