cl2: Scrape block profiles instead of mutex ones #2215
Conversation
In k/k, contention-profiling can be enabled using the --contention-profiling flag. However, this enables block profiling and not mutex profiling. CL2 as of now scrapes mutex profiles and not block profiles, because of which inspite of enabling contention profiling on control plane components, the profiles are still empty. Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com>
k8s-ci-robot
added
kind/bug
|
/assign @marseel |
| @@ -218,26 +218,26 @@ func (t *testMetrics) Execute(config *measurement.Config) ([]measurement.Summary | |||
| appendResults(&summaries, errList, summary, executeError(t.etcdCPUProfile.String(), action, err)) | |||
| summary, err = execute(t.etcdMemoryProfile, etcdGatherConfig) | |||
| appendResults(&summaries, errList, summary, executeError(t.etcdMemoryProfile.String(), action, err)) | |||
| summary, err = execute(t.etcdMutexProfile, etcdGatherConfig) | |||
There was a problem hiding this comment.
For Etcd I can see that mutex profiles are actually working, check https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1602710237461090304/artifacts/
There was a problem hiding this comment.
Yeah etcd actually enables both block and mutex:
https://github.com/etcd-io/etcd/blob/8e3fc380dcaf44c37bb5dc19f1832981225061cc/pkg/debugutil/pprof.go#L28 and https://github.com/etcd-io/etcd/blob/8e3fc380dcaf44c37bb5dc19f1832981225061cc/pkg/debugutil/pprof.go#L43
I think we may have to introduce a new type called BlockProfile, enable that for all control plane components and then use MutexProfile only for etcd if we want etcd mutex profiles
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: MadhavJivrajani, marseel The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
What type of PR is this?
/kind bug cleanup
What this PR does / why we need it:
In k/k, contention-profiling can be enabled using the --contention-profiling flag. However, this enables block profiling and not mutex profiling. CL2 as of now scrapes mutex profiles and not block profiles, because of which inspite of enabling contention profiling on control plane components, the profiles are still empty.
As seen here for the API Server,
--contention-profilingenables block profiling:https://github.com/kubernetes/kubernetes/blob/68f808e6db388b249dae8c6c9dfb078bc0947b49/staging/src/k8s.io/apiserver/pkg/server/config.go#L919-L920
and similarly for other components.
See slack discussion here.