Support DeleteSnapshot Retries in SnapshotWriter

[sam-heilbron]

Description

Enhance the DeleteSnapshot method of the SnapshotWriter to consume the same retryOptions that the WriteSnapshot operation uses

Context

Why is this necessary?

When deleting resources in bulk, we may delete a resource and while gloo is processing that deleting, we delete another resource that references it. This will result in Gloo rejecting the deletion of the second resource. In a perfect world, Gloo could handle these bulk updates. However, for now, we mirror how we handle writing resources in bulk in tests, we just retry the whole operation. This retry gives time for Gloo to process the changes, and eventually accept the proposed deletion.

How is this tested?

I wrote a kube2e test which configures a resource that does not exist. In the past, this would have thrown an error. We now swallow "does not exist" errors, to ensure we can retry operations. The test demonstrates that we successfully swallow these errors and continue deleting the snapshot.

This does NOT test that we retry correctly. However, this logic is identical to the writeSnapshot logic, so I decided it was ok to not have full test coverage. Also, if we see that flakes are still existing, then it will be a signal that there is a gap in the functionality.

Checklist:

• I included a concise, user-facing changelog (for details, see https://github.com/solo-io/go-utils/tree/main/changelogutils) which references the issue that is resolved.
• If I updated APIs (our protos) or helm values, I ran make -B install-go-tools generated-code to ensure there will be no code diff
• I followed guidelines laid out in the Gloo Edge contribution guide
• I opened a draft PR or added the work in progress label if my PR is not ready for review
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[solo-changelog-bot]

Issues linked to changelog:
https://github.com/solo-io/solo-projects/issues/5162

[github-actions]

Visit the preview URL for this PR (updated for commit 32355b1):

https://gloo-edge--pr8456-stability-snapshot-d-fdbcpb0n.web.app

_{(expires Tue, 18 Jul 2023 16:49:53 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}

[ashishb-solo]

this also looks like it should fix https://github.com/solo-io/solo-projects/issues/5156