helm post-delete hooks is not working with argocd as expected

[dsharma24101992]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
I have an helm application present in path https://github.com/dsharma24101992/helmHooksDemo/tree/main/deployments/helm-hooks-demo which has 2 helm hooks . First one is pre-install hook which will create the clusterrole update-namespaces(Please refer to file https://github.com/dsharma24101992/helmHooksDemo/blob/main/deployments/helm-hooks-demo/templates/pre-install-hooks-job.yaml and job helm-hooks-demo-create-clusterrole-pre-install-job) . second is a post-delete hook.as part of post-delete hook(https://github.com/dsharma24101992/helmHooksDemo/blob/main/deployments/helm-hooks-demo/templates/post-delete-hooks-job.yaml), I am deleting the clusterrole update-namespaces as part of job delete-clusterrole). my understanding is that post-delete hooks for clusterrole, clusterrolebinding,serviceAccount should be be created first and as soon as my job delete-clusterrole is completed, the clusterrole ,clusterrolebinding,serviceAccount which are created as part of post-delete hooks should be deleted. but I am getting the following error when post-delete hooks are executing in argoUI.

clusterroles.rbac.authorization.k8s.io "helm-hooks-demo-post-delete" already exists

I am doing the same steps via helm install and uninstall, everything is working fine. below are the uninstall logs using helm uninstall.

deesharma@dac0272-lcedt:~/Downloads$ helm uninstall helm-hooks-demo -n helm-hooks-demo --debug 
uninstall.go:102: [debug] uninstall: Deleting helm-hooks-demo
client.go:486: [debug] Starting delete for "allow-all-traffic" NetworkPolicy
client.go:142: [debug] creating 1 resource(s)
client.go:142: [debug] creating 1 resource(s)
client.go:142: [debug] creating 1 resource(s)
client.go:142: [debug] creating 1 resource(s)
client.go:712: [debug] Watching for changes to Job delete-clusterrole with timeout of 5m0s
client.go:740: [debug] Add/Modify event for delete-clusterrole: ADDED
client.go:779: [debug] delete-clusterrole: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for delete-clusterrole: MODIFIED
client.go:779: [debug] delete-clusterrole: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for delete-clusterrole: MODIFIED
client.go:779: [debug] delete-clusterrole: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for delete-clusterrole: MODIFIED
client.go:486: [debug] Starting delete for "helm-hooks-demo-post-delete" ClusterRole
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:486: [debug] Starting delete for "helm-hooks-demo-post-delete" ServiceAccount
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:486: [debug] Starting delete for "helm-hooks-demo-post-delete" ClusterRoleBinding
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:486: [debug] Starting delete for "delete-clusterrole" Job
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 
5m0s
uninstall.go:155: [debug] purge requested for helm-hooks-demo
release "helm-hooks-demo" uninstalled

I have attached the argoUI snapshot which I have taken after trigger application delete from argoUI.

helm-hooks-demo-UI-error during deletion

helm-hooks-demo-UI-install during application installation

In summary , the problem statement is why helm-hooks-demo-post-delete clusterrole is not getting deleted as soon as the post-delete job is completed.

To Reproduce

1. Git repo => https://github.com/dsharma24101992/helmHooksDemo.git
2. create the minikube cluster and install argo using command kubectl apply -n argocd https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.10/manifests/install.yaml
3. Configure the helm-hooks-demo argo application using these 2 commands
   1. kubectl apply -f https://raw.githubusercontent.com/dsharma24101992/helmHooksDemo/main/helm-hooks-argo-application.yaml
   2. kubectl apply -f https://raw.githubusercontent.com/dsharma24101992/helmHooksDemo/main/helm-hooks-configmap.yaml
4. Delete the application from argoUI using foreground-deletion propogation policy.
5. The issue should be reproduced and the application deletion will throw the error clusterroles.rbac.authorization.k8s.io "helm-hooks-demo-post-delete" already exists

Expected behavior

Screenshots

Version

argocd: v2.5.8+bbe870f
  BuildDate: 2023-01-25T16:17:49Z
  GitCommit: bbe870ff5904dd1cebeba6c5dcb7129ce7c2b5e2
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.5.8+bbe870f
  BuildDate: 2023-01-25T16:17:49Z
  GitCommit: bbe870ff5904dd1cebeba6c5dcb7129ce7c2b5e2
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: could not get kustomize version: exec: "kustomize": executable file not found in $PATH
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

[agaudreault]

Hey @dsharma24101992, can you add the relevant logs of your application for the application-controller component?

This might be related to #18270

[dsharma24101992]

@agaudreault Yes, I also think the issue is similar. Please refer to the application-controller logs. Note that, the logs I am pasting is from the moment I tried the delete of the helm-hooks-demo application.

argocd-application-controller-logs.txt

[agaudreault]

Closed by #18767. Feel free to reopen after you have tested with the code change if the bug is still happening.