Controller doesn't fully clean up removed resources

[tech1ndex]

Essentially as stated in the title, we have Terraform resources that we are applying via Flux.

In the log below you can see that the "destroy" and "destroyResourcesOnDeletion" is called successfully however once the resource is actually removed, it seems to be thrown into a failed reconciliation loop which results in some stale artifacts being left behind in Metrics and being reported in an "Unknown" state.

tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:56:52.480Z","msg":"calling plan ...","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:56:52.505Z","msg":"plan to destroy","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:56:57.710Z","msg":"plan: ok, found drift: true","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:56:57.753Z","msg":"save tfplan: ok","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:56:57.813Z","msg":"load tf plan: ok","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:57:00.997Z","msg":"apply: ok","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:57:00.997Z","msg":"inventory is disabled","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:57:01.022Z","msg":"finalizing destroyResourcesOnDeletion: ok","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:57:01.061Z","msg":"finalizing secrets: ok","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:57:01.087Z","msg":"clean up dir: ok","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:57:01.087Z","msg":"setting up terraform","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"error","ts":"2024-07-09T15:57:01.088Z","msg":"unable to update status before Terraform initialization","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z","error":"Terraform.infra.contrib.fluxcd.io \"5539da99-8111-4e42-8f23-0496e977a0b2\" not found"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"error","ts":"2024-07-09T15:57:01.088Z","msg":"error in terraform setup","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z","error":"Terraform.infra.contrib.fluxcd.io \"5539da99-8111-4e42-8f23-0496e977a0b2\" not found"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"info","ts":"2024-07-09T15:57:01.088Z","msg":"clean up dir: ok","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"error","ts":"2024-07-09T15:57:01.088Z","msg":"unable to update status after the reconciliation is complete","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","reconciliation-loop-id":"3a3b2f5b-74f0-4922-9832-af9b074d668e","start-time":"2024-07-09T15:56:34.691Z","error":"Terraform.infra.contrib.fluxcd.io \"5539da99-8111-4e42-8f23-0496e977a0b2\" not found"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"error","ts":"2024-07-09T15:57:06.107Z","msg":"Reconciler error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"bbdfadac-2ca3-48ef-bdb7-42cb25396149","error":"Terraform.infra.contrib.fluxcd.io \"5539da99-8111-4e42-8f23-0496e977a0b2\" not found"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"error","ts":"2024-07-09T15:57:06.107Z","msg":"Hit an error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"d9e12a45-d383-4129-8eb7-b3fd6c818ae0","reconciliation-loop-id":"398a88bd-10c8-48ce-8a8f-d9f1e668dd94","start-time":"2024-07-09T15:57:06.107Z","function":"TerraformReconciler.Reconcile","namespacedName":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"error":"Terraform.infra.contrib.fluxcd.io \"5539da99-8111-4e42-8f23-0496e977a0b2\" not found"}
tf-controller-655d5c74c7-mr7xl tf-controller {"level":"error","ts":"2024-07-09T15:57:06.113Z","msg":"Hit an error","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"namespace":"edge-workspace","name":"5539da99-8111-4e42-8f23-0496e977a0b2","reconcileID":"a975fe22-a460-4d93-9857-8b2278804552","reconciliation-loop-id":"fad918b4-ecd9-4053-8077-4de80eb8ceef","start-time":"2024-07-09T15:57:06.113Z","function":"TerraformReconciler.Reconcile","namespacedName":{"name":"5539da99-8111-4e42-8f23-0496e977a0b2","namespace":"edge-workspace"},"error":"Terraform.infra.contrib.fluxcd.io \"5539da99-8111-4e42-8f23-0496e977a0b2\" not found"}

[TarasLykhenko]

#1362 should solve this issue.

[greenu]

#1362 should solve this issue.

@TarasLykhenko Seems not resolved. I see deleted Terraform objects still remain in metrics in Unknown status.
Using latest tofu-controller main-cb032d47