Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not allow EC2 instance ID NotFound to succeed tagging #674

Merged
merged 1 commit into from
Oct 14, 2023
Merged

Do not allow EC2 instance ID NotFound to succeed tagging #674

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions pkg/providers/v1/tags.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,10 +322,6 @@ func (c *Cloud) TagResource(resourceID string, tags map[string]string) error {
output, err := c.ec2.CreateTags(request)

if err != nil {
if isAWSErrorInstanceNotFound(err) {
klog.Infof("Couldn't find resource when trying to tag it hence skipping it, %v", err)
return nil
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if we return this error instead of silencing it, the workItem will be re-queued and presumably we'll successfully tag the instance after the API becomes consistent?

How long did we observe that to take in this scenario?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, that's the intention 👍

from the logs we have for this event, CCM executes the tagging work item almost exactly when the instance launches. given that when experimenting through cli

ID=$(aws ec2 run-instances --image-id ami-07d07d65c47e5aa90 --instance-type t2.micro --query Instances[0].InstanceId --output text)
aws ec2 create-tags --resources $ID --tags Key=test,Value=value
aws ec2 describe-tags --filters Name=resource-id,Values=$ID

you pretty much can't encounter the issue, i think any amount of retry in place would fix the issue

klog.Errorf("Error occurred trying to tag resources, %v", err)
return err
}
Expand All @@ -346,6 +342,8 @@ func (c *Cloud) UntagResource(resourceID string, tags map[string]string) error {
output, err := c.ec2.DeleteTags(request)

if err != nil {
// An instance not found should not fail the untagging workflow as it
// would for tagging, since the target state is already reached.
if isAWSErrorInstanceNotFound(err) {
klog.Infof("Couldn't find resource when trying to untag it hence skipping it, %v", err)
return nil
Expand Down
5 changes: 3 additions & 2 deletions pkg/providers/v1/tags_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
"testing"

"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/stretchr/testify/assert"
"k8s.io/klog/v2"
Expand Down Expand Up @@ -268,8 +269,8 @@ func TestTagResource(t *testing.T) {
{
name: "tagging failed due to resource not found error",
instanceID: "i-not-found",
err: nil,
expectedMessage: "Couldn't find resource when trying to tag it hence skipping it",
err: awserr.New("InvalidInstanceID.NotFound", "Instance not found", nil),
expectedMessage: "Error occurred trying to tag resources",
},
}

Expand Down