cdk synth/deploy: tries to describe default CDKToolkit bootstrap stack even when a qualifier and custom toolkit stack name are used

[polarizeme]

Describe the bug

If your cdk.json contains the following config line:
"@aws-cdk/core:bootstrapQualifier": "example-qualifier"

And you bootstrapped an account with:
cdk bootstrap aws://<accountID>/us-east-1 --toolkit-stack-name "some-custom-stack-name" --qualifier "example-qualifier" --cloudformation-execution-policies "<policyARN>"

When you do a cdk synth or a cdk deploy, the action will still look for the default CDKToolkit bootstrap/toolkit stack, which is entirely unnecessary in a situation where it never existed in the first place. In some cases, like ours, it completely breaks the ability to deploy anything.

Expected Behavior

If you're using a custom qualifier and a custom stack name, cdk commands should NOT be trying to run anything against a stack that never existed, even though it's the default; the action should check for a custom qualifier FIRST, use that to determine the bootstrap/toolkit stack name, and THEN run the necessary actions.

Current Behavior

Synth start example:

[14:23:01] [trace] SdkProvider#withAwsCliCompatibleDefaults()
[14:23:01] Toolkit stack: CDKToolkit

Deploy start example:

[13:58:12] [trace] SdkProvider#withAwsCliCompatibleDefaults()
[13:58:12] Toolkit stack: CDKToolkit

Deploy ending example:

 deploying... [1/1]
[13:58:27] [trace] SdkProvider#resolveEnvironment()
[13:58:27] [trace]   SdkProvider#defaultAccount()
[13:58:27] [trace] SdkProvider#baseCredentialsPartition()
[13:58:27] [trace]   SdkProvider#resolveEnvironment()
[13:58:27] [trace]   SdkProvider#obtainBaseCredentials()
[13:58:27] [trace]     SdkProvider#defaultAccount()
[13:58:27] [trace]     SdkProvider#defaultCredentials()
[13:58:27] [trace]   SDK#currentAccount()
[13:58:27] [trace]     SDK#forceCredentialRetrieval()
[13:58:27] Retrieved account ID 303699070191 from disk cache
[13:58:27] [trace] SDK#cloudFormation()
[13:58:27] [trace]   SDK#wrapServiceErrorHandling()
[13:58:27] Waiting for stack CDKToolkit to finish creating or updating...
[13:58:27] [AWS cloudformation 400 0.406s 0 retries] describeStacks({ StackName: 'CDKToolkit' })
[13:58:27] [trace] SDK#makeDetailedException()
[13:58:27] Call failed: describeStacks({"StackName":"CDKToolkit"}) => Stack with id CDKToolkit does not exist (code=ValidationError)
[13:58:27] Stack CDKToolkit does not exist
[13:58:27] The environment aws://<accountID>/us-east-1 doesn't have the CDK toolkit stack (CDKToolkit) installed. Use cdk bootstrap "aws://<accountID>/us-east-1" to setup your environment for use with the toolkit.
[13:58:27] [trace] SDK#ssm()
[13:58:27] [trace]   SDK#wrapServiceErrorHandling()
[13:58:28] [AWS ssm 200 0.408s 0 retries] getParameter({ Name: '/cdk-bootstrap/example-qualifier/version' })

Now, from what I can gather, in most cases it doesn't seem this will matter. The process will determine that there's a custom stack name (likely based on the qualifier and the SSM parameter for it that points to the cfn stack specified when bootstrapping), BUT there are cases, like ours, where it completely breaks the ability to deploy anything.

We have a large organization with lots of roles that are guarded by a combination of SCPs and Permission Boundaries. In this specific case, we have multiple teams bootstrapping each account w/ their own custom qualifier and custom stack name so that we don't step on each other's toes. There are things in place to ensure that folks from team A cannot touch a bootstrap stack created by Team B, and this extends to the role(s) created by team A's bootstrap. And specifically, we have rules in place so that no one can do anything with CDKToolkit as a cfn stack so that we stay entirely away from that default stack name.

And now we arrive at our issue. Whenever we try to run a deploy, we get this error:

[20:17:23] Waiting for stack CDKToolkit to finish creating or updating...
--
423 | [20:17:23] [AWS cloudformation 403 0.05s 0 retries] describeStacks({ StackName: 'CDKToolkit' })
424 | [20:17:23] [trace] SDK#makeDetailedException()
425 | [20:17:23] Call failed: describeStacks({"StackName":"CDKToolkit"}) => User: arn:aws:sts::<accountID>:assumed-role/<cdk deploy role> is not authorized to perform: cloudformation:DescribeStacks on resource: arn:aws:cloudformation:us-east-1:<accountID>:stack/CDKToolkit with an explicit deny in a service control policy (code=AccessDenied)

Which makes it seem like the SCP is the issue, except for the fact that we literally have no stack with this name to begin with, so cdk should NOT be trying to describe it in the first place.

Reproduction Steps

1. Bootstrap an account with:
   cdk bootstrap aws://<accountID>/us-east-1 --toolkit-stack-name "some-custom-stack-name" --qualifier "example-qualifier" --cloudformation-execution-policies "<policyARN>"

2. Make sure the context in your cdk.json contains the following config line:
   "@aws-cdk/core:bootstrapQualifier": "example-qualifier"

3. try to use cdk deploy to the bootstrapped account; use the -vvv flag if you want to see the debug output that contains the attempts to use the default toolkit stack name.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.86.0

Framework Version

No response

Node.js Version

18

OS

MacOS 13.2

Language

Python

Language Version

3.11.3

Other information

No response

[pahud]

Thank you for the report. We'll discuss this with the team.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.