(stepfunctions-tasks): AutoTerminationPolicy feature is not supported by Step Functions

[frederic-schmaljohann]

What is the problem?

I'm not sure but the following seems to me like a bug:

When I try to add the AutoTerminationPolicy in tasks.EmrCreateCluster, get the following error while deploying:

Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The field "AutoTerminationPolicy" is not supported by Step Functions

I get the same error when I try to the AutoTerminationPolicy in the console when creating a cluster via Step Functions, which makes me wonder how this feature could even work.

This feature was introduced in cdk 1.132.0 with #16976

Thank you and regards
Frederic

Reproduction Steps

1. Define a State machine
2. Add a step to create an EMR cluster with tasks.EmrCreateCluster and AutoTerminationPolicy
3. Deploy

What did you expect to happen?

A state machine which starts an EMR cluster with AutoTerminationPolicy

What actually happened?

The schema of the state machine is invalid.

CDK CLI Version

1.132.0

Framework Version

No response

Node.js Version

17.0.1

OS

Mac OS

Language

Python

Language Version

3.9.7

Other information

No response

[kaizencc]

@frederic-schmaljohann sorry for the confusion! You are correct, the docs say "Step Functions does not terminate an Amazon EMR cluster automatically if execution is stopped." This was a miss on my part, I will look to revert the previous PR.

[frederic-schmaljohann]

@kaizen3031593 Ok, thank you. I don't think that the explanation is correct though, AutoTerminationPolicy is a feature provided by EMR and StepFunctions wouldn't need to terminate the cluster. However, passing the termination policy doesn't seem to be supported by StepFunctions. The documentation is not 100% correct here in my opinion: https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html

[kaizencc]

I agree, the documentation is not good here. I took the above sentence to maybe mean that Stepfunctions does not allow automatic termination to EMR clusters but even that is a stretch...

I believe you, I'm going to have to get to the bottom of this and most likely deprecate the property.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.