use 'moonbeam' prefix when emitting prometheus metrics

[nbaztec]

What does it do?

Configures the prometheus registry to use moonbeam prefix when emitting metrics. The metrics would continue to use the chain id as a value for the label chain.

Based on similar polkadot feature here.

What important points reviewers should know?

Existing (grafana) dashboards will break since the metrics are renamed (eg: moonbeam_substrate_block_height).

Existing metrics do not use a prefix and hence are emitted as is. This change will prepend moonbeam_ to all metric names that are emitted by the client: substrate_block_height -> moonbeam_substrate_block_height.

All consumers of the metrics (e.g. Grafana) need to use the prefixed metric names going forward.

Is there something left for follow-up PRs?

(Manual) Integration test must be performed to ascertain that the runtime's prometheus registry override does not inadvertently override prefix for metrics emitted outside the runtime.
Confirmed locally with prometheus.

What alternative implementations were considered?

Are there relevant PRs or issues in other repositories (Substrate, Polkadot, Frontier, Cumulus)?

What value does it bring to the blockchain users?

Clients can easily distinguish the emitted metrics by the moonbeam application. Distinct chains can be filtered on the chain label.

[librelois]

thank @nbaztec, this is an excellent first contribution :)

This is a breaking change for ops, so we'll have to remember to warn them when we release the client v0.22!

cc @purestakeoskar @Mitchell-Grimes, so that you already have it in your head.

[nbaztec]

There's another existing issue for Export chain name in prometheus metrics, this can easily be done as part of this PR by adding the chain_spec.id() as a global const label (all metrics would contain it).

I'm just not sure if the following is the desired behavior for that issue:

moonbeam_my_counter{chain="moonriver", value=1.0}
moonbeam_my_counter{chain="moonbeam", value=2.0}

Any thoughts?

[librelois]

@artkaseman, what do you think?

[Mitchell-Grimes]

There's another existing issue for Export chain name in prometheus metrics, this can easily be done as part of this PR by adding the chain_spec.id() as a global const label (all metrics would contain it).

I'm just not sure if the following is the desired behavior for that issue:

moonbeam_my_counter{chain="moonriver", value=1.0}
moonbeam_my_counter{chain="moonbeam", value=2.0}

Any thoughts?

This would be how Polkadot now separates its metrics, we do some magic in our backend so we will be able to handle incoming changes for our dashboards not to break. But really looking forward to any new additional metrics, especially Moonbeam specific ones 🙂

[artkaseman]

The original use case for the chain id is actually no longer valid. It would be a nice to have but would this greatly increase cardinality if it's included in every metric? @PSjoe

[PSjoe]

The original use case for the chain id is actually no longer valid. It would be a nice to have but would this greatly increase cardinality if it's included in every metric? @PSjoe

I'm afraid I don't follow exactly what's going on here. If we're talking about adding a new label like chain="moonriver" to all the metrics, we all ready to this with our Prometheus rules, so there would be no change in cardinality. Others might appreciate it though.

[nbaztec]

I agree with @PSjoe, the cardinality is already present and shouldn't impact too much if a collector is only scraping a few distinct chains.

[librelois]

@nbaztec When the Pr is approved and the CI is green you can merge directly, unless there is a code freeze when a release is coming.