velero restore create failed to restore pv only

[jhuisss]

Describe the bug

I installed velero 1.9.2 and use velero-plugin-for-vsphere to handle pv backup and restore. But velero restore create failed to restore pv only.

To Reproduce
[Steps to reproduce the behavior]

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   REASON   AGE
pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5   50Mi       RWO            Delete           Bound    nginx-example/nginx-logs   vsphere-csi             27m

# kubectl get pvc -A
NAMESPACE       NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-example   nginx-logs   Bound    pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5   50Mi       RWO            vsphere-csi    27m

# velero backup create backup-pv-2 --include-resources pv

# velero backup describe backup-pv-2
Name:         backup-pv-2
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.23.8+vmware.2
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=23

Phase:  Completed

Errors:    0
Warnings:  1

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        pv
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2022-10-28 09:35:53 +0000 UTC
Completed:  2022-10-28 09:36:06 +0000 UTC

Expiration:  2022-11-27 09:35:53 +0000 UTC

Total items to be backed up:  1
Items backed up:              1

Velero-Native Snapshots: <none included>

# velero backup logs backup-pv-2 --insecure-skip-tls-verify
...
time="2022-10-28T09:36:06Z" level=info msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/backup-pv-2 logSource="pkg/backup/item_backupper.go:462" name=pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5 namespace= persistentVolume=pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5 resource=persistentvolumes volumeSnapshotLocation=default
time="2022-10-28T09:36:06Z" level=info msg="Persistent volume is not a supported volume type for snapshots, skipping." backup=velero/backup-pv-2 logSource="pkg/backup/item_backupper.go:473" name=pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5 namespace= persistentVolume=pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5 resource=persistentvolumes
time="2022-10-28T09:36:06Z" level=info msg="Backed up 1 items out of an estimated total of 1 (estimate will change throughout the backup)" backup=velero/backup-pv-2 logSource="pkg/backup/backup.go:380" name=pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5 namespace= progress= resource=persistentvolumes
time="2022-10-28T09:36:06Z" level=info msg="Backed up a total of 1 items" backup=velero/backup-pv-2 logSource="pkg/backup/backup.go:405" progress=

Here the msg "Persistent volume is not a supported volume type for snapshots, skipping." is pretty weird, is this expected?

Then I forced remove the pv from the workload cluster:

# kubectl get pv
No resources found

# kubectl get pvc -A
NAMESPACE       NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-example   nginx-logs   Lost     pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5   0                         vsphere-csi    30m

Then I try to restore the pv from the backup:

# velero restore create --from-backup backup-pv-2
# velero restore describe backup-pv-2-20221028094736
Name:         backup-pv-2-20221028094736
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       Completed
Total items to be restored:  1
Items restored:              1

Started:    2022-10-28 09:47:37 +0000 UTC
Completed:  2022-10-28 09:47:37 +0000 UTC

Backup:  backup-pv-2

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Existing Resource Policy:   <none>

Preserve Service NodePorts:  auto

# velero restore logs backup-pv-2-20221028094736 --insecure-skip-tls-verify
...
time="2022-10-28T09:47:37Z" level=info msg="Getting client for /v1, Kind=PersistentVolume" logSource="pkg/restore/restore.go:882" restore=velero/backup-pv-2-20221028094736
time="2022-10-28T09:47:37Z" level=info msg="Dynamically re-provisioning persistent volume because it doesn't have a snapshot and its reclaim policy is Delete." logSource="pkg/restore/restore.go:1099" restore=velero/backup-pv-2-20221028094736
time="2022-10-28T09:47:37Z" level=info msg="Restored 1 items out of an estimated total of 1 (estimate will change throughout the restore)" logSource="pkg/restore/restore.go:669" name=pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5 namespace= progress= resource=persistentvolumes restore=velero/backup-pv-2-20221028094736
...
time="2022-10-28T09:47:37Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:518" restore=velero/backup-pv-2-20221028094736

But the pv did't show up:

# kubectl get pv
No resources found
# kubectl get pvc -A
NAMESPACE       NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-example   nginx-logs   Lost     pvc-d0d3599a-6d4d-4818-abb8-c7c78123b4d5   0                         vsphere-csi    33m

Expected behavior

The pv should be restored.

Troubleshooting Information

[Please refer to the Troubleshooting page and collect the required information]

Screenshots

[If applicable, add screenshots to help explain your problem.]

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue]

[jhuisss]

bundle-2022-11-02-12-18-49.tar.gz
bundle-2022-11-02-12-19-05.tar.gz
The velero debug --backup/restore logbundle are attached, thanks.

[Busesz]

vSphere Backup NFC is checked on a VMkernel port?