Updating and adding a timeout to the cleanup cluster step #7727
Conversation
| @@ -52,5 +52,14 @@ for ns in $namespaces; do | |||
| break | |||
| fi | |||
| echo "deleting namespaces: $ns" | |||
|
|
|||
| # Remove finalizers from the namespace if present | |||
| kubectl get namespace $ns -o json | jq '.spec.finalizers = []' >temp-namespace.json | |||
There was a problem hiding this comment.
Interestingly, UCP and Applications RP Pods were not healthy, and this caused finalizers to not finalize in some of the resources in some of the namespaces. And then this caused those namespaces to not be deleted.
UCP Pod Down -> Finalizers in some resources -> Namespaces couldn't be deleted -> Clean up step got stuck -> Long Running Workflow failed back-to-back
There was a problem hiding this comment.
Do you have logs from the failing pods? or from the Kubernetes controller-manager pod?
| @@ -515,6 +515,7 @@ jobs: | |||
| --yes --verbose | |||
| - name: Clean up cluster | |||
| if: always() | |||
| timeout-minutes: 60 | |||
There was a problem hiding this comment.
Added timeout to clean up step.
There was a problem hiding this comment.
What's the goal of the timeout? That we'd be notified if we hit it?
There was a problem hiding this comment.
Default is 6 hours, so fail faster?
There was a problem hiding this comment.
Reformatted the files
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #7727 +/- ##
==========================================
- Coverage 61.00% 60.99% -0.01%
==========================================
Files 520 520
Lines 27010 27010
==========================================
- Hits 16478 16476 -2
- Misses 9080 9081 +1
- Partials 1452 1453 +1 ☔ View full report in Codecov by Sentry. |
Radius functional test overview
Click here to see the list of tools in the current test run
Test Status⌛ Building Radius and pushing container images for functional tests... |
Signed-off-by: ytimocin <ytimocin@microsoft.com>
ytimocin
force-pushed
the
ytimocin/fixLRWorkflow
branch
from
65113ae
to
d8a3fd0
Compare
ytimocin
requested a deployment
to
functional-tests
GitHub Actions
Waiting
| kubectl delete namespace $ns --ignore-not-found=true | ||
| done | ||
|
|
||
| # Cleanup temporary file | ||
| rm -f temp-namespace.json |
There was a problem hiding this comment.
if an error occurs above/script exists, is it possible that file is not cleaned up ?
|
|
||
| # Remove finalizers from the namespace if present | ||
| kubectl get namespace $ns -o json | jq '.spec.finalizers = []' >temp-namespace.json | ||
| kubectl replace --raw "/api/v1/namespaces/$ns/finalize" -f ./temp-namespace.json |
There was a problem hiding this comment.
Does --force help here? https://kubernetes.io/docs/reference/kubectl/generated/kubectl_delete/
Just thinking it might be easier if it works.
I'm also curious what the finalizers are. We have to be a little bit careful with this, because certain types in Kubernetes will manage cloud resources. If we delete the Kubernetes resource without running the finalizers, then the cloud resources leak.
Description
Updating and adding a timeout to the cleanup cluster step.
Type of change