Additional metrics

[adam-cattermole]

Changes

• Added liveness_latency histogram metric, cache_size gauge and batcher_size gauge
• Made the datastore_partitioned metric always present
• Ensured that we only record datastore_latency metrics if the timing data is present
• For redis_cached we aggregate the datastore span calls by the batcher flushes
• Refactored to store the metrics layer consumer function in a Box, enabling different consumer functions to be provided
• Added a batcher_flush_size histogram
• Added an eviction_listener to the cache to allow us to decrement the cache_size as well as to monitor the evicted_pending_writes in a histogram
• Still thinking on the implementation.. but added a counter_overshoot histogram for when we add_from_authority and which pushes us over the counter max val

[alexsnaps]

diff --git a/limitador-server/src/metrics.rs b/limitador-server/src/metrics.rs
index b09444b..d41df57 100644
--- a/limitador-server/src/metrics.rs
+++ b/limitador-server/src/metrics.rs
@@ -1,6 +1,5 @@
 use std::collections::HashMap;
 use std::ops;
-use std::sync::Arc;
 use std::time::{Duration, Instant};
 use tracing::span::{Attributes, Id};
 use tracing::Subscriber;
@@ -73,12 +72,12 @@ impl SpanState {
 }
 
 pub struct MetricsGroup {
-    consumer: Arc<fn(Timings)>,
+    consumer: Box<fn(Timings)>,
     records: Vec<String>,
 }
 
 impl MetricsGroup {
-    pub fn new(consumer: Arc<fn(Timings)>, records: Vec<String>) -> Self {
+    pub fn new(consumer: Box<fn(Timings)>, records: Vec<String>) -> Self {
         Self { consumer, records }
     }
 }
@@ -99,7 +98,7 @@ impl MetricsLayer {
         let rec = records.iter().map(|r| r.to_string()).collect();
         self.groups
             .entry(aggregate.to_string())
-            .or_insert(MetricsGroup::new(Arc::new(consumer), rec));
+            .or_insert(MetricsGroup::new(Box::new(consumer), rec));
         self
     }
 }

Works on my computer™ 😉

[adam-cattermole]

Addressed that now @alexsnaps - I think I must have been confusing things with the original implementation where I was having difficulty with the boxed function

[alexsnaps]

You can't call that pending_writes... while the value got evicted from the cache, it's still in the queue! But now, when it'll get consumed, it won't have pending writes anymore... You can "just" call no_pending_writes on it. if it had none, then you can ignore this eviction, but if we start evicting entries with pending writes, it probably means you need to review your cache size

[adam-cattermole]

Right gotcha, so instead of a histogram of the number of pending writes it'll be a (metrics) counter that increments in the case that we evict an entry with no_pending_writes().not()