Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exponential backoff in the LifetimeWatcher does not send an error to doneCh if we reach the lease expiration while failing to renew #28611

Open
jSherz opened this issue Oct 7, 2024 · 1 comment · May be fixed by #28616
Labels
auth/k8s bug Used to indicate a potential bug core/api ecosystem

Comments

@jSherz
Copy link

jSherz commented Oct 7, 2024

Describe the bug

In #26868, the backoff package was upgraded to V4. Part of the difference between V3 and V4 of backoff is that the backoff.ExponentialBackOff struct now takes a "Stop" field, indicating the value to return when we should no longer retry the operation.

In the constructor they provide (backoff.NewExpontentialBackOff - https://github.com/cenkalti/backoff/blob/v4/exponential.go#L94), the Stop value is set to the constant in the package, a duration of -1 second. The Vault code however directly initialises the struct (https://github.com/hashicorp/vault/blob/c8e6169d5dbc82d99d904e468852902de98ebfd0/api/lifetime_watcher.go) and thus we end up with a duration of zero seconds for the "Stop" field.

The lifetime watcher later checks the returned value from the backoff to see if it's backoff.Stop. It never is, as the ExponentialBackoff we've initialised returns a time.Duration of zero seconds.

To Reproduce

  1. Try to use the lifetime watcher and have it error out when we've unsuccessfully retried to renew the lease and the lease duration has expired.

Expected behavior

r.doneCh should receive an error.

Environment:

	github.com/hashicorp/vault/api v1.15.0
	github.com/hashicorp/vault/api/auth/kubernetes v0.8.0
@jSherz jSherz changed the title Exponential backoff in the LifetimeWatcher is broken Exponential backoff in the LifetimeWatcher does not send an error to doneCh if we reach the lease expiration while failing to renew Oct 7, 2024
@jSherz
Copy link
Author

jSherz commented Oct 7, 2024

I had a go at a fix in #28616 - please let me know what you think. Is the current behaviour the expected one?

@heatherezell heatherezell added bug Used to indicate a potential bug core/api ecosystem auth/k8s labels Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auth/k8s bug Used to indicate a potential bug core/api ecosystem
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants