[Telemetry] track and warn event loop delays thresholds

[Bamieh]

Summary

Part of a larger work to measure platform performance and ease debugging performance issues (#63848)

In 7.14 we started sending hourly updated event loop delays histogram (#101580). This helps us investigate average delays our customers have, percentiles, etc.

This PR warns users when the event loop delay exceeds a configurable threshold duration ops. eventLoopDelayThreshold

By default this duration is 350ms logged once every 30 seconds as long as the delay is still above that target.

Once we have a representative sample from the reported delays histogram we can adjust this default to be more meaningful and closer to real world cases.

metrics.ops already reports collected_at and all the ecs object for further debuggablity around the logs.

Implementation direction

This is another approach to implementing the event loop threshold (original draft PR: #103478)

Instead of using the ops.metrics implementation I used the the event loop delays histogram. The current ops metics implementation does not really capture event loop delays as it only captures the delay in the immediate loop when the measurement is made.

The perf.monitorEventLoopDelay() here tracks the delays over time and not only on collection. This way we really capture delays and spikes. I also added some telemetry around these spikes to report them back to our cluster along the full histogram for diagnosis. which is not possible inside core at the moment without a lot of piping. I prefer this approach, let me know what you think.

Notes

I've experimented with using event loop utilization (ELU) but realized it serves a different purpose than the original intention of this PR (draft #103477)

Related: #98673
Closes: #96192

[Bamieh]

This is another approach to implementing the event loop threshold (original draft PR: #103478)

Instead of using the ops.metrics implementation I used the the event loop delays histogram. The current ops metics implementation does not really capture event loop delays as it only captures the delay in the immediate loop when the measurement is made.

The perf.monitorEventLoopDelay() tracks the delays over time and not only during the collection. This way we really capture delays and spikes. I also added some telemetry around these spikes to report them back to our cluster along the full histogram for diagnosis. which is not possible inside core at the moment without a lot of piping.

I like this approach more. let me know what you think. cc @joshdover

[joshdover]

Just one nit, but otherwise LGTM. Totally agree that this is going to be a much more helpful, stable, and accurate way of warning the admin of an issue. I do worry that this warning isn't highly actionable though, but would signals to support that they may have some scaling issues with Task Manager.

I do wonder if we should consider linking to our documentation on scaling task manager in production: https://www.elastic.co/guide/en/kibana/master/task-manager-production-considerations.html#_deployment_considerations

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: f83f9c1
• Storybooks Preview

Metrics [docs]

✅ unchanged

History

• 💚 Build #135022 succeeded 0c47654
• 💔 Build #134960 failed 821372e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.