Additional Metrics collected and exposed via prometheus

[gnunicorn]

This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

• system load average (over one, five and 15min)
• the TCP connection state of the process (lsof), refs Track additional metrics around resource consumption #5304
• number of tokio threads
• block import/validation times*
• number of known forks
• count items in each unbounded queue (with internal unbounded channels)
• number of file descriptors opened by this process (*nix only at this point)
• number of system threads (*nix only at this point)

refs #4679

---

* While this would be a good case for a histogram, the way this works in promotheus doesn't fit our use case for two reasons: 1. it assumes a set of pre-known buckets the value should fall under; 2. it locks and calculates this bucket right there and then – which leads to a significant performance drop (50% bps here). Thus, this uses an internal ringbuffered queue (based off this future-rs patch) to track them without locking and then calculate count, average and medians when we tick.

[tomaka]

Are we aware that the content of primitives/utils/src/channels is not going to be maintainable by us?
Is this code destined to stay there long term?

[gnunicorn]

@tomaka hopefully not, as the FIXME said. Unfortunately neither std nor futures-rs exposes the internally used queue, otherwise I would have taking a much slimmer approach than copying the entire folder over from the PR – we only want the non-blocking ringbuffer feature here after all. Either way, once they accept and release or reject that PR, this part is prone for being overthrown.

[gnunicorn]

because of timing and priority, I postponed

• block production times
• count of block productions failure

for a future PR.

[gnunicorn]

Removed the Block Import Tracker and internally used ring buffer. Also applied the other requested changes. Please review again.

[expenses]

Some nitpicks, but these are great changes overall!

[mxinden]

Thanks for reducing the scope of this pull request. As a general wish I would prefer multiple small pull requests instead of one large pull request whenever the changes are not directly related. E.g. instrumenting our unbounded channels is not directly related to the service metric refactorings. Having multiple pull requests makes reviewing easier and does not have one controversial change block an easy one.

I have added a couple of comments inline.

Would you mind taking a look at the additional newlines introduced in this pull request? As far as I remember we e.g. don't start an impl with a newline before the first function. Please correct me if I am wrong.

[mxinden]

The last round is just a set of alignment issues, typos or unnecessary newlines.

Thanks for all the work you have put into this!

[mxinden]

Comments don't need to block merging.