vSphere with Tanzu - Unable to restore PVC: StorageClassName is not set

[majla1]

Describe the bug
On vSphere with Tanzu in the guest cluster a restore does not set StorageClass for pv,pvc and changes reclaim policy from Delete to Retain.
Next attempt for backup/restore of the previously restored volumes returns: Failed to create a CloneFromSnapshot: Failed for PVC because StorageClassName is not set.

To Reproduce
Login to target cluster

kubectl-vsphere login --server supervisor.tanzu --tanzu-kubernetes-cluster-name dev-cl05 --tanzu-kubernetes-cluster-namespace development

Create simple workload with pv in namespace nginx-pvc

kubectl apply -f nginx_w_volume.yaml

Get pv,pvc info

kubectl get pv,pvc -n nginx-pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS                REASON   AGE
persistentvolume/pvc-2130b522-d5b4-441e-b41d-94d9339c590f   1Gi        RWO            Delete           Bound    nginx-pvc/nginx-pvc   tanzu-storage-policy-gold            13m

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
persistentvolumeclaim/nginx-pvc   Bound    pvc-2130b522-d5b4-441e-b41d-94d9339c590f   1Gi        RWO            tanzu-storage-policy-gold   13m

Backup the nginx-pvc namespace

velero backup create nginx-pvc-dev-cl05-01 --include-namespaces nginx-pvc

List backups

[root@tanzuclient velero]# velero backup get
NAME                    STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
nginx-dev-cl05          Completed   0        0          2023-02-22 18:10:42 +0100 CET   28d       default            <none>
nginx-pvc-dev-cl05-01   Completed   0        0          2023-02-24 10:08:27 +0100 CET   29d       default            <none>

Delete the workload

kubectl delete -f nginx_w_volume.yaml

Check if pv,pvc were deleted

kubectl get pv,pvc -n nginx-pvc
No resources found

Restore the workload

velero restore create restore-nginx-pvc-dev-cl05-01 --from-backup nginx-pvc-dev-cl05-01

List restores

velero restore get
NAME                            BACKUP                  STATUS      STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
restore-nginx-pvc-dev-cl05-01   nginx-pvc-dev-cl05-01   Completed   2023-02-24 10:20:34 +0100 CET   2023-02-24 10:20:53 +0100 CET   0        0          2023-02-24 10:20:34 +0100 CET   <none>

The workload was restored but pc,pvc has empty Storage Class

kubectl get pv,pvc -n nginx-pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
persistentvolume/pvc-fabfd226-1c09-45bf-90be-e4c5c2f11e8f   1Gi        RWO            Retain           Bound    nginx-pvc/nginx-pvc                           4m31s

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/nginx-pvc   Bound    pvc-fabfd226-1c09-45bf-90be-e4c5c2f11e8f   1Gi        RWO                           4m31s

The corresponding pvc on supervisor has the correct Storage Class

kubectl get persistentvolume/pvc-fabfd226-1c09-45bf-90be-e4c5c2f11e8f -o yaml|grep volumeHandle
          f:volumeHandle: {}
    volumeHandle: ngin-6d27d84c-c122-4422-8d11-b2c76bf88dee

kubectl config use-context development
Switched to context "development".

kubectl get pvc -n development |grep ngin-6d27d84c-c122-4422-8d11-b2c76bf88dee
ngin-6d27d84c-c122-4422-8d11-b2c76bf88dee                                   Bound    pvc-a671139e-3542-4c71-bbc2-1324aa23a503   1Gi        RWO            tanzu-storage-policy-gold   106m

Expected behavior
Persistent volumes should be restored with Storage Class

Troubleshooting Information

vCenter version: 7.0.3.20395099
velero version

Client:
        Version: v1.8.1
        Git commit: 18ee078dffd9345df610e0ca9f61b31124e93f50
Server:
        Version: v1.8.1

velero client config get features
features: <NOT SET>
Plugins:
velero-plugin-for-vsphere:v1.4.0
velero-plugin-for-aws:v1.5.1

Kubernetes cluster flavor: vSphere with Tanzu

kubectl version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2+vmware.2", GitCommit:"098524fa6b545cf2ec2290259f34d59005dee622", GitTreeState:"clean", BuildDate:"2021-04-28T20:17:41Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.9+vmware.1", GitCommit:"21eeb4527eefb360eb251addc358cea6997e8335", GitTreeState:"clean", BuildDate:"2022-05-04T00:13:17Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Screenshots

[If applicable, add screenshots to help explain your problem.]

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue]
velero_logs.tar.gz

[deepakkinni]

Could i have the velero logs specific to the backup and restore?

[majla1]

Could i have the velero logs specific to the backup and restore?

velero_logs.tar.gz contains backup driver and velero logs from TKG cluster. Please let me know which logs you need. Thank you.