Add convergence tracker failure on timeout

kube-burner-workload/README.md

@@ -333,6 +333,13 @@ and also may be reused and improved by the same platform as a part of this frame
 
 Every platform may have its own README.
 
+### Comparing different platforms
+
+To ensure results for different platform are comparable, set up the convergence tracker logic to be as similar as possible,
+all timeouts and variables defining successful test run should be the same.
+Cluster-specific parameters, like resource quotas, enables services (e.g. observability), nodes configurations may also
+affect the results.
+
 ## Tracking the end of the test
 
 `CONVERGENCE_TRACKER` env variable enables `convergence-tracker` job.
@@ -347,7 +354,7 @@ before deleting the workload.
 
 ## Running
 
-1. Install kube-burner v1.9.0+
+1. Install kube-burner v1.9.4+
 
     1.1 You can download kube-burner from https://github.com/cloud-bulldozer/kube-burner/releases
 

kube-burner-workload/convergence_waiter.sh

@@ -3,9 +3,14 @@
 TIME_SPENT=0
 TIMEOUT=$((CONVERGENCE_TIMEOUT + CONVERGENCE_PERIOD))
 while [ $TIME_SPENT -le "$TIMEOUT" ]; do
-  # failure will return 1 because of the "echo FAILED| wc -l"
-  PODS_COUNT=$( (kubectl get pods -n convergence-tracker-0 --no-headers || echo FAILED) | grep -c -v Completed)
-  if [ "$PODS_COUNT" -eq 0 ]; then
+  FAILED_COUNT=$(kubectl get pods -n convergence-tracker-0 --field-selector status.phase=Failed -o name | wc -l)
+  if [ "$FAILED_COUNT" -ne 0 ]; then
+    echo "ERROR: convergence tracker pod reported failure"
+    kubectl get pods -n convergence-tracker-0 --field-selector status.phase=Failed -o name
+    exit 1
+  fi
+  RUNNING_COUNT=$(kubectl get pods -n convergence-tracker-0 --field-selector status.phase!=Succeeded -o name | wc -l)
+  if [ "$RUNNING_COUNT" -eq 0 ]; then
     echo "DONE"
     exit 0
   fi

kube-burner-workload/openshift/README.md

@@ -14,6 +14,11 @@
 `kube-burner init -m ./openshift/metrics.yml -c ./network-policy.yaml -u https://$(oc get route prometheus-k8s -n openshift-monitoring -o jsonpath="{.spec.host}") --log-level=debug --token=$(oc create token prometheus-k8s -n openshift-monitoring)`
 8. When the test finishes, metrics should be collected by the ES_SERVER
 
+## Finding the limit
+
+To automate finding the limit, [test_limit.sh](./test_limit.sh) script may be used.
+It can run multiple iterations increasing the number of network policies until test fails.
+It waits for full cleanup after every iteration to ensure the cluster is ready for the next one.
 
 ## Metrics and Dashboards
 

kube-burner-workload/openshift/convergence_tracker.yml

@@ -12,10 +12,16 @@ spec:
       labelSelector:
         matchLabels:
           app: convergence-tracker
+  tolerations:
+    - key: "node-role.kubernetes.io/master"
+      operator: "Exists"
   volumes:
     - name: openvswitch
       hostPath:
         path: /var/run/openvswitch
+    - name: ovn
+      hostPath:
+        path: /var/run/ovn/
     - name: ovn-ic
       hostPath:
         path: /var/run/ovn-ic/
@@ -42,8 +48,10 @@ spec:
           mountPath: /var/run/openvswitch
         - name: host-var-log-ovs
           mountPath: /var/log/openvswitch
-        - name: ovn-ic
+        - name: ovn
           mountPath: /var/run/ovn
+        - name: ovn-ic
+          mountPath: /var/run/ovn-ic
         - name: pod-logs
           mountPath: /var/log/pods
       env: