[Telemetry] Instrument APM around snapshot telemetry generation

[afharo]

We want to minimize the impact the telemetry collection has on production clusters. To have more visibility around them, let's instrument some APM transactions and spans when running each collector to measure and identify the most costly actions performed by the usage collectors.

Relevant piece of code:

kibana/src/plugins/usage_collection/server/collector/collector_set.ts

Lines 219 to 242 in 7e7f862

	public bulkFetch = async (
	esClient: ElasticsearchClient,
	soClient: SavedObjectsClientContract,
	collectors: Map<string, AnyCollector> = this.collectors
	) => {
	this.logger.debug(`Getting ready collectors`);
	const getMarks = createPerformanceObsHook();
	const { readyCollectors, nonReadyCollectorTypes, timedOutCollectorsTypes } =
	await this.getReadyCollectors(collectors);
	// freeze object to prevent collectors from mutating it.
	const context = Object.freeze({ esClient, soClient });
	const fetchExecutions = await Promise.all(
	readyCollectors.map(async (collector) => {
	const wrappedPromise = perfTimerify(
	`fetch_${collector.type}`,
	async () => await this.fetchCollector(collector, context)
	);
	return await wrappedPromise();
	})
	);
	const durationMarks = getMarks();

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

@lizozom, given your experience with APM instrumentation, it'd be great if you could advise us on the best way to instrument these: one transaction usage_collection and one span per collector? multiple transactions?

Bear in mind collectors run in parallel, does that affect the spans/transactions (ie: will they close each other)?

[rudolf]

APM can help us see why the wall time is slow (e.g. because there's several request running serially) but it cannot answer the question "why is ... making Kibana slow". When it comes to telemetry the wall time is not the problem, the problem is that it sometimes blocks the CPU and causes slow event loops.

[afharo]

When it comes to telemetry the wall time is not the problem, the problem is that it sometimes blocks the CPU and causes slow event loops.

I'd say it's both: we received claims that it could also affect ES' performance (#93770), even though we've demonstrated most of the time it's just highlighting another underlying problem.

Re event loop delays, it looks like APM also monitors that (elastic/apm-agent-nodejs#1053).