[ClusterLoader2] Prometheus metrics corrupted

[jprzychodzen]

While running tests with ClusterLoader2, sometimes created snapshot lacks data. Probably, this is related to creating a snapshot of a running disk.

For small tests (~15 minutes long) this can result in missing metrics for specific tests run.

[jkaniuk]

I've just bumped into that as well.
It happens relatively frequently for 100 nodes tests - 25% of runs out of random 8 runs?

Prometheus logs when data is missing entirely:

tsdb msg="last page of the wal is torn, filling it with zeros" segment=/prometheus-db/wal/00000000
tsdb msg="deleting all segments newer than corrupted segment" segment=0

Missing entirely:

• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1225352227909013505
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-networkpolicies/1223218919838322690

Looks good:

• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1225338388027019266 (24m of data)
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1225309197101436929 (23m of data)
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability/1225294850157449222 (25m of data)
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-networkpolicies/1225307936805359618 (43m of data)
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-networkpolicies/1225126496062935042 (43m of data)
• https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-scalability-networkpolicies/1225035640324755464 (30m of data - duration of Prometheus stack running)

~20-40m of data expected:

W0206 08:56:09.385] I0206 08:56:09.380792   13158 prometheus.go:133] Setting up prometheus stack
...
W0206 09:18:25.438] I0206 09:18:25.437107   13158 prometheus.go:178] Tearing down prometheus stack
another run:
W0206 06:55:49.818] I0206 06:55:49.817762   13305 prometheus.go:166] Prometheus stack set up successfully
...
W0206 07:35:12.182] I0206 07:35:12.174525   13305 prometheus.go:178] Tearing down prometheus stack

/area clusterloader
/priority critical-urgent