clusterloader2/testing/load: Adjust kube-state-metrics latency tests

[lilic]

Enable kube-state-metrics to be run in the presubmit load tests.
/hold

[lilic]

FYI @mrueg @brancz @tariq1890 this just enables it temporarily so we see the results in the 100 node presubmit tests. 🎉

[lilic]

@wojtek-t, this is the PR as discussed earlier. Thanks!

[lilic]

Update: Seems like its failing to find the service object for kube-state-metrics, looking into it.

[mrueg]

@lilic F0323 14:49:28.989150 14091 clusterloader.go:291] Error while setting up prometheus stack: parsing error: template: :34: unexpected {{end}} could it be related to 5a6de85#diff-95ffc56dd3788da48793718c29a759d29cd9a8fa1242e9ea8bcb6dd672f0b6e4R34 ? There are two {{end}} statements that might cause it

[lilic]

@mrueg I took that from

perf-tests/clusterloader2/pkg/prometheus/manifests/prometheus-prometheus.yaml

Lines 18 to 28 in 5a6de85

	requests:
	cpu: {{AddInt 200 (MultiplyInt 500 (DivideInt .Nodes 1000))}}m
	{{if $PROMETHEUS_SCRAPE_KUBELETS}}
	memory: 10Gi
	{{else}}
	# Start with 2Gi and add 2Gi for each 1K nodes.
	memory: {{MultiplyInt 2 (AddInt 1 (DivideInt .Nodes 1000))}}Gi
	{{end}}
	limits:
	{{if $PROMETHEUS_SCRAPE_KUBELETS}}
	memory: 10Gi

but you might be right, there is no need for end as we are not using the if. Good catch!

[mrueg]

Is it valid in k8s to have a non-numerical targetPort? Is it possible, that could be the reason it can't find the service? 5a6de85#diff-38d60ed161d961ff44471bdbfbdfef4c3a3c78d2baaec3bc4ba21cfc9c2cc5a2R17

[lilic]

@mrueg it worked correctly for

pull-perf-tests-clusterloader2 — Job succeeded.

We can see the results of the latency test https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/perf-tests/1774/pull-perf-tests-clusterloader2/1374398775702851584/artifacts/KubeStateMetricsLatency_load_2021-03-23T16:58:30Z.json

About the non-numerical ports -> https://kubernetes.io/docs/concepts/services-networking/service/#multi-port-services

For example, the names 123-abc and web are valid, but 123_abc and -web are not.

[mrueg]

@mrueg it worked correctly for

pull-perf-tests-clusterloader2 — Job succeeded.

We can see the results of the latency test https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/perf-tests/1774/pull-perf-tests-clusterloader2/1374398775702851584/artifacts/KubeStateMetricsLatency_load_2021-03-23T16:58:30Z.json

About the non-numerical ports -> https://kubernetes.io/docs/concepts/services-networking/service/#multi-port-services

For example, the names 123-abc and web are valid, but 123_abc and -web are not.

Thanks! I didn't know this was possible 👍

[lilic]

Update on this, KSM uses the "in-cluster" config, which is incompatible with kubemark this is why they are failing. Thanks @wojtek-t ! I will remove KSM from running under the kubemark job.

Added code to disable this KSM test when running in kubemark env. If all is green will take out the enabled by default parts and we can proceed with the other changes.

[lilic]

kubmark passed as it was skipped and the other load test passed and these are the results for kube-state-metrics latency loads results for the 100 node cluster:

First run:

{
"Perc50": 243181818,
"Perc90": 456730769,
"Perc99": 725000000
}

Second run:

{
"Perc50": 259615384,
"Perc90": 475000000,
"Perc99": 906666666
}

Will remove the other commits that enable the runs by default and remove the WIP.

[lilic]

/hold cancel

@wojtek-t do you might having a look, thank you!

I will do the refactor you suggested in a separate PR if that's okay for you?

[wojtek-t]

We're trying to be positive in those (e.g. talk about features not about disabling something).

So instead of that, we should make it more like:

SupportKubeStateMetrics

Given that our tests run on GCE, it's probably enough to enable that just for gce cloud provider for now.

[wojtek-t]

/lgtm
/approve

Thanks!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lilic, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• clusterloader2/OWNERS [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment