Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: referencing arrays/lists when using crossRegionReferences fails with "SerializationException: Start of list found where not expected" #31655

Open
1 task
nathanielks opened this issue Oct 4, 2024 · 1 comment
Labels
@aws-cdk/core Related to core CDK functionality bug This issue is a bug. effort/medium Medium work item – several days of effort p1

Comments

@nathanielks
Copy link

nathanielks commented Oct 4, 2024

Describe the bug

When creating stacks with crossRegionReferences enabled, and the stack exposes a list/array value that is to be read in another region, the Custom::CrossRegionExportWriter resource will fail during creation with the error "SerializationException: Start of list found where not expected". Reading string values works fine, but if it's an array/list, it will fail.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

I expect to be able to reference lists/arrays from a stack in another region like I would strings.

Current Behavior

The Custom::CrossRegionExportWriter resource fails during creation with the error "SerializationException: Start of list found where not expected":

BugVPCStack-us-west-2
BugVPCStack-us-west-2: deploying... [1/4]
Bug-VPC1-Requests-VPC2:  start: Building bc980b3b9776ac14eeed5eeecde033423deae6a101589e745c39d5e53e119d87:current_account-us-west-2
Bug-VPC1-Requests-VPC2:  success: Built bc980b3b9776ac14eeed5eeecde033423deae6a101589e745c39d5e53e119d87:current_account-us-west-2
Bug-VPC2-Accepts-VPC1:  start: Building a7bb53c2fb4a92a7dcc01a0863f4145f4161926cdb3d34becc9fe4a65ce54566:current_account-us-east-1
Bug-VPC2-Accepts-VPC1:  success: Built a7bb53c2fb4a92a7dcc01a0863f4145f4161926cdb3d34becc9fe4a65ce54566:current_account-us-east-1
BugVPCStack-us-west-2: creating CloudFormation changeset...
3:14:45 PM | UPDATE_FAILED        | Custom::CrossRegionExportWriter       | ExportsWriteruseast10F67B507DDE2E818
Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of list found where not expected
at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
at async Promise.all (index 0)
at async u (/var/task/index.js:3:330) (RequestId: f443b2b6-c5f3-49e6-98ce-eec9f39ef19d)


 ❌  BugVPCStack-us-west-2 failed: Error: The stack named BugVPCStack-us-west-2 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of list found where not expected
    at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async Promise.all (index 0)
    at async u (/var/task/index.js:3:330) (RequestId: f443b2b6-c5f3-49e6-98ce-eec9f39ef19d)
    at FullCloudFormationDeployment.monitorDeployment (/Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:458:10567)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:461:200334)
    at async /Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:461:181756

 ❌ Deployment failed: Error: The stack named BugVPCStack-us-west-2 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of list found where not expected
    at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async Promise.all (index 0)
    at async u (/var/task/index.js:3:330) (RequestId: f443b2b6-c5f3-49e6-98ce-eec9f39ef19d)
    at FullCloudFormationDeployment.monitorDeployment (/Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:458:10567)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Object.deployStack2 [as deployStack] (/Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:461:200334)
    at async /Users/aang/Code/Work/repos/core/cdk/Vpc-Peering/node_modules/aws-cdk/lib/index.js:461:181756

The stack named BugVPCStack-us-west-2 failed to deploy: UPDATE_ROLLBACK_COMPLETE: Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of list found where not expected
    at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async Promise.all (index 0)
    at async u (/var/task/index.js:3:330) (RequestId: f443b2b6-c5f3-49e6-98ce-eec9f39ef19d)

Reproduction Steps

By default, the app will be created in a way that does not trigger the bug. To deploy the app as-is:

cdk bootstrap
cdk deploy --all --app 'node app.ts'

This should create the stacks successfully. Then run another deployment with a context variable:

cdk deploy --all --app 'node app.ts' --context readIPv6=true

This will instruct the app to reference list values in a cross-region manager and trigger the bug.

The bug occurs whether you create the stack with or without the context variable set; that is to say, if you create the stack with the context variable, the bug will also be triggered in that circumstance.

app.ts:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';

interface VpcStackProps extends cdk.StackProps {
  readonly cidr: string;
}

class VpcStack extends cdk.Stack {
  public readonly vpcId: string;
  public readonly vpcCidrIpV4: string;
  public readonly vpcCidrIpV6: string;
  public readonly vpcCidrIpV6List: string[];

  public constructor(scope: cdk.App, id: string, props: VpcStackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'Bug VPC', {
      ipAddresses: ec2.IpAddresses.cidr(props.cidr),
      ipProtocol: ec2.IpProtocol.DUAL_STACK,
      createInternetGateway: false,
      maxAzs: 3,
      subnetConfiguration: [
        {
          cidrMask: 28,
          name: 'private',
          subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
        }
      ]
    })

    this.vpcId = vpc.vpcId
    this.vpcCidrIpV4 = vpc.vpcCidrBlock

    // Both the selected value and the original list value will trigger the bug
    this.vpcCidrIpV6 = cdk.Fn.select(0, vpc.vpcIpv6CidrBlocks)
    this.vpcCidrIpV6List = vpc.vpcIpv6CidrBlocks
  }
}

function createVpcStack(region: string, cidr: string) {
  const stack = new VpcStack(app, `BugVPCStack-${region}`, {
    env: { region },
    crossRegionReferences: true,
    cidr
  })

  return stack
}

const ec2 = cdk.aws_ec2
const app = new cdk.App();

const vpc1Stack = createVpcStack('us-west-2', '10.16.0.0/16')
const vpc2Stack = createVpcStack('us-east-1', '10.17.0.0/16')

const requesterStack = new cdk.Stack(app, `Bug-VPC1-Requests-VPC2`, {
  env: { region: vpc1Stack.region },
  crossRegionReferences: true
})

const readIPv6 = app.node.tryGetContext('readIPv6')
const values = [
  vpc1Stack.vpcId!,
  vpc1Stack.vpcCidrIpV4!,
  vpc2Stack.vpcId!,
  vpc2Stack.vpcCidrIpV4!,
  ...(readIPv6 
    ? [
      vpc1Stack.vpcCidrIpV6!,
      vpc2Stack.vpcCidrIpV6!,
    ]
    : [])
]

new cdk.aws_ssm.StringListParameter(requesterStack, 'sourceVpc', {
  stringListValue: values,
})

if (readIPv6) new cdk.aws_ssm.StringListParameter(requesterStack, 'sourceVpcIpv6', {
  stringListValue: vpc2Stack.vpcCidrIpV6List!,
})

const accepterStack = new cdk.Stack(app, `Bug-VPC2-Accepts-VPC1`, {
  env: { region: vpc2Stack.region },
  crossRegionReferences: true
})

new cdk.aws_ssm.StringListParameter(accepterStack, 'sourceVpc', {
  stringListValue: values,
})

if (readIPv6) new cdk.aws_ssm.StringListParameter(accepterStack, 'sourceVpcIpv6', {
  stringListValue: vpc1Stack.vpcCidrIpV6List!,
})

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.158.0

Framework Version

No response

Node.js Version

20.15.0

OS

macOS 14.6.1

Language

TypeScript

Language Version

5.5.4

Other information

No response

@nathanielks nathanielks added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 4, 2024
@github-actions github-actions bot added the @aws-cdk/core Related to core CDK functionality label Oct 4, 2024
@pahud
Copy link
Contributor

pahud commented Oct 5, 2024

Reproducible. This seems a bug to me.

Let me simplify it with this PoC

VpcStack

export interface VpcStackProps extends cdk.StackProps {
  readonly cidr?: string;
}
export class VpcStack extends Stack {
  public readonly vpcId: string;
  public readonly vpcCidrIpV4: string;
  public readonly vpcCidrIpV6: string;
  public readonly vpcCidrIpV6List: string[];
  constructor(scope: Construct, id: string, props: VpcStackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, 'Bug VPC', {
      ipAddresses: ec2.IpAddresses.cidr('10.16.0.0/16'),
      ipProtocol: ec2.IpProtocol.DUAL_STACK,
      createInternetGateway: false,
      maxAzs: 3,
      subnetConfiguration: [
        {
          cidrMask: 28,
          name: 'private',
          subnetType: ec2.SubnetType.PRIVATE_ISOLATED,
        }
      ]
    })

    this.vpcId = vpc.vpcId
    this.vpcCidrIpV4 = vpc.vpcCidrBlock

    this.vpcCidrIpV6 = cdk.Fn.select(0, vpc.vpcIpv6CidrBlocks);
    new CfnOutput(this, 'vpcCidrIpV6', { value: this.vpcCidrIpV6  })
  }
}

It's fine when just deploy the vpc stack

new VpcStack(app, 'vpc-stack', { env });
Outputs:
vpc-stack.vpcCidrIpV6 = 2600:1f18:601f:ec00::/56

Now if we define a consumer stack

export interface ConsumerStackProps extends StackProps {
  readonly vpcCidrIpV6: string;
}

export class ConsumerStack extends Stack {
  constructor(scope: Construct, id: string, props: ConsumerStackProps) {
    super(scope, id, props);

    new CfnOutput(this, 'MyListOutput', { value: props.vpcCidrIpV6 })
  }
}

And enable crossRegionReferences:

const stack = new VpcStack(app, 'vpc-stack', { env, crossRegionReferences: true });

new ConsumerStack(app, 'consumer-stack', { env: {
    account: process.env.CDK_DEFAULT_ACCOUNT,
    region: 'us-west-2',
} , vpcCidrIpV6: stack.vpcCidrIpV6, crossRegionReferences: true })

The writer CR would fail with SerializationException:

11:03:42 PM | CREATE_FAILED | Custom::CrossRegionExportWriter | ExportsWriteruswes...0/Resource
/Default
Received response status [FAILED] from custom resource. Message returned: SerializationException: Start of l
ist found where not expected
at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.j
s:840:20)
at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-ssm/dist-cjs/index.js:7021:14)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:165:18
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38

Making this a p1.

@pahud pahud added p1 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Oct 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/core Related to core CDK functionality bug This issue is a bug. effort/medium Medium work item – several days of effort p1
Projects
None yet
Development

No branches or pull requests

2 participants