Skip to content

Commit

Permalink
Create a small docs page on custom active series trackers (#1491)
Browse files Browse the repository at this point in the history
Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>
Co-authored-by: Marco Pracucci <marco@pracucci.com>
Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Co-authored-by: Christopher Moyer <35463610+chri2547@users.noreply.github.com>
  • Loading branch information
4 people authored Mar 16, 2022
1 parent a1f87b7 commit 6199ee5
Showing 1 changed file with 38 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "Configuring custom active series trackers"
menuTitle: "Configuring custom active series trackers"
description: "Use the custom tracker to count the number of active series on an ingester."
weight: 55
---

# Configuring custom active series trackers

You can use the custom tracker feature to count the number of active series on an ingester that match a particular label pattern.

The label pattern to match against is specified using the `-ingester.active-series-custom-trackers` CLI flag (or its respective YAML configuration option). Each custom tracker is defined as a key-value pair, where the key is the name of the tracker and the value is the label matcher. Both the key and the value are type `<string>`.

The following example configures a custom tracker to count the active series coming from `dev` and `prod` namespaces for each tenant.

```yaml
active_series_custom_trackers:
dev: '{namespace=~"dev-.*"}'
prod: '{namespace=~"prod-.*"}'
```
If you configure a custom tracker for an ingester, the ingester exposes a `cortex_ingester_active_series_custom_tracker` gauge metric on its [/metrics endpoint](({{< relref "../reference-http-api#metrics" >}})).

Each custom tracker counts the active series matching its label pattern on a per-tenant basis, which means that each custom tracker generates as many as `# of tenants` series with metric name `cortex_ingester_active_series_custom_tracker`. To reduce the cardinality of this metric, only custom trackers that have matched at least one series are exposed on the metric, and they are removed if they become `0`.

Series with metric name `cortex_ingester_active_series_custom_tracker` have two labels applied: `name` and `user`. The value of the `name` label is the name of the custom tracker specified in the configuration. The value of the `user` label is the tenant-id for which the series count applies.

Assume two custom trackers are configured as in the example above, and that your Grafana Mimir cluster has three tenants: `tenant_1`, `tenant_2`, and `tenant_with_only_prod_metrics`. Assume all series within `tenant_with_only_prod_metrics` have labels that match the pattern `{namespace=~"prod-.*"}` and none that match `{namespace=~"dev-.*"}`.

In this example, the following output appears when the `/metrics` endpoint for the ingester component is scraped:

```
cortex_ingester_active_series_custom_tracker{name="dev", user="tenant_1"}
cortex_ingester_active_series_custom_tracker{name="prod", user="tenant_2"}
cortex_ingester_active_series_custom_tracker{name="prod", user="tenant_with_only_prod_metrics"}
```

> **Note:** The custom active series trackers are exposed on each ingester. To understand the count of active series matching a particular label pattern in your Grafana Mimir cluster at a global level, you must collect and sum this metric across all ingesters. If you're running Grafana Mimir with a `replication_factor` > 1, you must also adjust for the fact that the same series will be replicated `RF` times across your ingesters.

0 comments on commit 6199ee5

Please sign in to comment.