Do not allow EC2 instance ID NotFound to succeed tagging

[ndbaker1]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Removes the graceful handling of InvalidInstanceID.NotFound error when attempting to tag an ec2 instance that has not fully come up. This has caused an issue where we've seen the tagging controller misleadingly exit successfully, not actually tagging the instance, and does not re-queue the item to (ideally) execute again once the instance becomes visible.

example log feed:

tags.go:326] Couldn't find resource when trying to tag it hence skipping it, InvalidInstanceID.NotFound: The instance ID 'i-***' does not exist status code: 400, request id: ***
tagging_controller.go:299] Successfully tagged i-*** with map[aws:eks:cluster-name:***]. Labeling the nodes with tagging controller labels now.
tagging_controller.go:305] Successfully labeled node ip-***.compute.internal with map[k8s.io/cloud-provider-aws:***].

This behavior does satisfy the untag action, since removing the tag from a non-existing instance is a no-op, so no changes need to be made there.

Its worth mentioning the initial PR to gracefully handle this (#448) aimed to fix all cases discussed in issue #444 where the untracked InvalidInstanceID.NotFound errors were valid failure modes in the context of instance termination.

Which issue(s) this PR fixes: N/A

Special notes for your reviewer: N/A

Does this PR introduce a user-facing change?:

NONE

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: ndbaker1 / name: Nick Baker (805b07f)

[k8s-ci-robot]

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Welcome @ndbaker1!

It looks like this is your first PR to kubernetes/cloud-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/cloud-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @ndbaker1. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[cartermckinnon]

So if we return this error instead of silencing it, the workItem will be re-queued and presumably we'll successfully tag the instance after the API becomes consistent?

How long did we observe that to take in this scenario?

[ndbaker1]

yep, that's the intention 👍

from the logs we have for this event, CCM executes the tagging work item almost exactly when the instance launches. given that when experimenting through cli

ID=$(aws ec2 run-instances --image-id ami-07d07d65c47e5aa90 --instance-type t2.micro --query Instances[0].InstanceId --output text)
aws ec2 create-tags --resources $ID --tags Key=test,Value=value
aws ec2 describe-tags --filters Name=resource-id,Values=$ID

you pretty much can't encounter the issue, i think any amount of retry in place would fix the issue

[cartermckinnon]

/ok-to-test

[cartermckinnon]

Change looks fine to me. IIUC, silencing this error in #448 was just an optimization; if there is an errant Node in the API, we'll try to tag it n times, but there's no correctness issue per se?

This will show up in our error metrics, but I think that's appropriate.

[cartermckinnon]

/retest

[hakman]

/release-note-none

[hakman]

Looks like a good change for better reliability. Thanks @ndbaker1!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment