Upgrade "csi-secret-store-provider-driver" failed: pre-upgrade hooks failed: * job secrets-store-csi-driver-upgrade-crds failed: BackoffLimitExceeded

[utkarsh222739]

Have you

• Read Troubleshooting Guide
• Searched on GitHub issues and Discussions

What steps did you take and what happened:
When I tried running below command -

helm upgrade --install --debug --wait --timeout 10m0s csi-secret-store-provider-driver csi-secret-store-provider-driver/ --namespace kube-system
I am getting error -

Upgrade "csi-secret-store-provider-driver" failed: pre-upgrade hooks failed: 1 error occurred: * job secrets-store-csi-driver-upgrade-crds failed: BackoffLimitExceeded

When I saw the logs with below command -

kubectl logs job/secrets-store-csi-driver-upgrade-crds -n kube-system

I found the below error-

Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.4.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.4.0\"},\"creationTimestamp\":null,\"name\":\"secretproviderclasses.secrets-store.csi.x-k8s.io\"},\"spec\":{\"group\":\"secrets-store.csi.x-k8s.io\",\"names\":{\"kind\":\"SecretProviderClass\",\"listKind\":\"SecretProviderClassList\",\"plural\":\"secretproviderclasses\",\"singular\":\"secretproviderclass\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1alpha1\",\"schema\":{\"openAPIV3Schema\":{\"description\":\"SecretProviderClass is the Schema for the secretproviderclasses API\",\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"spec\":{\"description\":\"SecretProviderClassSpec defines the desired state of SecretProviderClass\",\"properties\":{\"parameters\":{\"additionalProperties\":{\"type\":\"string\"},\"description\":\"Configuration for specific provider\",\"type\":\"object\"},\"provider\":{\"description\":\"Configuration for provider name\",\"type\":\"string\"},\"secretObjects\":{\"items\":{\"description\":\"SecretObject defines the desired state of synced K8s secret objects\",\"properties\":{\"annotations\":{\"additionalProperties\":{\"type\":\"string\"},\"description\":\"annotations of k8s secret object\",\"type\":\"object\"},\"data\":{\"items\":{\"description\":\"SecretObjectData defines the desired state of synced K8s secret object data\",\"properties\":{\"key\":{\"description\":\"data field to populate\",\"type\":\"string\"},\"objectName\":{\"description\":\"name of the object to sync\",\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"labels\":{\"additionalProperties\":{\"type\":\"string\"},\"description\":\"labels of K8s secret object\",\"type\":\"object\"},\"secretName\":{\"description\":\"name of the K8s secret object\",\"type\":\"string\"},\"type\":{\"description\":\"type of K8s secret object\",\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"}},\"type\":\"object\"},\"status\":{\"description\":\"SecretProviderClassStatus defines the observed state of SecretProviderClass\",\"properties\":{\"byPod\":{\"items\":{\"description\":\"ByPodStatus defines the state of SecretProviderClass as seen by an individual controller\",\"properties\":{\"id\":{\"description\":\"id of the pod that wrote the status\",\"type\":\"string\"},\"namespace\":{\"description\":\"namespace of the pod that wrote the status\",\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":[],\"storedVersions\":[]}}\n"}},"spec":{"versions":[{"name":"v1alpha1","schema":{"openAPIV3Schema":{"description":"SecretProviderClass is the Schema for the secretproviderclasses API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"SecretProviderClassSpec defines the desired state of SecretProviderClass","properties":{"parameters":{"additionalProperties":{"type":"string"},"description":"Configuration for specific provider","type":"object"},"provider":{"description":"Configuration for provider name","type":"string"},"secretObjects":{"items":{"description":"SecretObject defines the desired state of synced K8s secret objects","properties":{"annotations":{"additionalProperties":{"type":"string"},"description":"annotations of k8s secret object","type":"object"},"data":{"items":{"description":"SecretObjectData defines the desired state of synced K8s secret object data","properties":{"key":{"description":"data field to populate","type":"string"},"objectName":{"description":"name of the object to sync","type":"string"}},"type":"object"},"type":"array"},"labels":{"additionalProperties":{"type":"string"},"description":"labels of K8s secret object","type":"object"},"secretName":{"description":"name of the K8s secret object","type":"string"},"type":{"description":"type of K8s secret object","type":"string"}},"type":"object"},"type":"array"}},"type":"object"},"status":{"description":"SecretProviderClassStatus defines the observed state of SecretProviderClass","properties":{"byPod":{"items":{"description":"ByPodStatus defines the state of SecretProviderClass as seen by an individual controller","properties":{"id":{"description":"id of the pod that wrote the status","type":"string"},"namespace":{"description":"namespace of the pod that wrote the status","type":"string"}},"type":"object"},"type":"array"}},"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "secretproviderclasses.secrets-store.csi.x-k8s.io", Namespace: ""
for: "crds/secrets-store.csi.x-k8s.io_secretproviderclasses.yaml": CustomResourceDefinition.apiextensions.k8s.io "secretproviderclasses.secrets-store.csi.x-k8s.io" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"controller-gen.kubebuilder.io/version":"v0.4.0","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{\"controller-gen.kubebuilder.io/version\":\"v0.4.0\"},\"creationTimestamp\":null,\"name\":\"secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io\"},\"spec\":{\"group\":\"secrets-store.csi.x-k8s.io\",\"names\":{\"kind\":\"SecretProviderClassPodStatus\",\"listKind\":\"SecretProviderClassPodStatusList\",\"plural\":\"secretproviderclasspodstatuses\",\"singular\":\"secretproviderclasspodstatus\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1alpha1\",\"schema\":{\"openAPIV3Schema\":{\"description\":\"SecretProviderClassPodStatus is the Schema for the secretproviderclassespodstatus API\",\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"status\":{\"description\":\"SecretProviderClassPodStatusStatus defines the observed state of SecretProviderClassPodStatus\",\"properties\":{\"mounted\":{\"type\":\"boolean\"},\"objects\":{\"items\":{\"description\":\"SecretProviderClassObject defines the object fetched from external secrets store\",\"properties\":{\"id\":{\"type\":\"string\"},\"version\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"podName\":{\"type\":\"string\"},\"secretProviderClassName\":{\"type\":\"string\"},\"targetPath\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":[],\"storedVersions\":[]}}\n"}},"spec":{"versions":[{"name":"v1alpha1","schema":{"openAPIV3Schema":{"description":"SecretProviderClassPodStatus is the Schema for the secretproviderclassespodstatus API","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"status":{"description":"SecretProviderClassPodStatusStatus defines the observed state of SecretProviderClassPodStatus","properties":{"mounted":{"type":"boolean"},"objects":{"items":{"description":"SecretProviderClassObject defines the object fetched from external secrets store","properties":{"id":{"type":"string"},"version":{"type":"string"}},"type":"object"},"type":"array"},"podName":{"type":"string"},"secretProviderClassName":{"type":"string"},"targetPath":{"type":"string"}},"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io", Namespace: ""
for: "crds/secrets-store.csi.x-k8s.io_secretproviderclasspodstatuses.yaml": CustomResourceDefinition.apiextensions.k8s.io "secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions```

This behavior started happening when I upgraded to aks version 1.28.9.

**What did you expect to happen:**
We expected no errors. Same hem chart and configuration was working with aks version 1.27.9.

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Which access mode did you use to access the Azure Key Vault instance:**
[e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity]


**Environment:**

- Secrets Store CSI Driver version: (use the image tag): 0.2.0
- Azure Key Vault provider version: (use the image tag):
- Kubernetes version: (use `kubectl version` and `kubectl get nodes -o wide`): 1.28.9
- Cluster type: (e.g. AKS, aks-engine, etc): AKS
- Installation method: ([Helm](https://azure.github.io/secrets-store-csi-driver-provider-azure/docs/getting-started/installation/#deployment-using-helm) , [Deployment yamls](https://azure.github.io/secrets-store-csi-driver-provider-azure/docs/getting-started/installation/#using-deployment-yamls), [AKS managed add-on](https://docs.microsoft.com/en-us/azure/aks/csi-secrets-store-driver)):