-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Velero backs up only metadata and not the data itself (pvc data) #7938
Comments
@xing-yang |
@blackpiglet @xing-yang |
The vSphere plugin issue needs expertise from @xing-yang. I have one suggestion too. Please remove the CSI plugin from the environment, although this may not be the cause of your issue. |
@blackpiglet |
+1 |
We have further investigation:
Run the following script
The upload passed successfully:
Try the same on a Windows host:
It's probably an issue with minio's windows flavor because of a limitation in the NTFS file system (which doesn't support colon ':' in filenames): Here is the code that verifies it in Minio: |
@xing-yang @blackpiglet |
@devopsInterviews |
@blackpiglet |
Not sure whether you are using a commercial version of the vSphere k8s environment, e.g. TKGs, TKGm, or TKGI. |
Currently we do not use any commercial version and we depend on the open source community. |
@devopsInterviews |
@blackpiglet |
https://github.com/vmware-tanzu/velero-plugin-for-vsphere I suggest checking that the repository's active contributor. |
Thank you @blackpiglet. @dsu-igeek @xing-yang @lipingxue It looks like you are some of the active maintainers of the repo. |
Any update? thoughts? |
The same problem is occurring to me as well. |
We have not done any qualifications on windows so it is not officially supported yet. I'll have to add this to our backlog. |
Hi @xing-yang, thanks for the answer. |
@xing-yang , Any update ? Could you please let me know when you intend to start working on this bug? |
Hi all, We are experiencing issues with backing up and restoring data on our vanilla Kubernetes cluster running on vSphere 7.3 to a remote MinIO S3 server using Velero, the vSphere plugin, and the AWS plugin. We are seeking your assistance to resolve this issue.
Environment Details:
Kubernetes Version: Vanilla Kubernetes Cluster (1.26.1)
vSphere Version: 7.3
Backup Solution: Velero
velero version: 1.13.2
Plugins Used:
velero/velero-plugin-for-aws
1.6.1
vsphereveleroplugin/velero-plugin-for-vsphere
1.4.2
velero/velero-plugin-for-csi
0.7.1
Object Storage: MinIO S3 on a remote server
Current Setup:
We have deployed Velero with the following pods in the velero namespace:
Issue Details:
Backup Process: We are able to initiate backups using Velero, and the metadata (e.g., namespaces, services, pods, PVC definitions) appears to be backed up correctly.
Restore Process: During the restore process, all the Kubernetes objects (services, pods, PVCs, etc.) are restored successfully. However, the actual data inside the PVCs is not restored. The PVCs remain empty.
Observations:
Metadata Backup: The metadata is correctly migrated and visible after restoration.
Data Absence: The data within the PVCs is not restored, resulting in empty PVCs even though the PVC objects themselves are recreated.
Pod Logs: We have examined the logs of the datamgr-for-vsphere-plugin pods and observed errors related to invalid object names during the upload process to MinIO.
Log Excerpt from datamgr-for-vsphere-plugin Pod:
Log Excerpt from Restore Command:
Our Understanding of the Backup and Restore Process:
Backup Process:
Velero initiates a backup process, capturing both metadata and data snapshots.
The metadata is successfully backed up and stored in MinIO.
Data snapshots (IVDs) are taken but fail to upload to MinIO due to invalid object names.
Restore Process:
Velero restores the metadata correctly.
Attempts to restore data from MinIO fail, leading to empty PVCs.
Installation Command:
Here is the command we used to install Velero:
We seek your guidance on the following points:
Resolution for Invalid Object Names: How can we ensure that object names generated during the snapshot process are valid and compatible with MinIO? We don't have control over them as far as I know.
Ensuring Data Backup and Restore: Steps we can take to ensure that both metadata and data are correctly backed up and restored.
We appreciate any insights or recommendations you can provide to help us resolve this issue.
Thank you for your support.
Best regards !
The text was updated successfully, but these errors were encountered: