This repo contains tools used for interacting with Cortex.
- cortextool: Interacts with user-facing Cortex APIs and backend storage components
- chunktool: Interacts with chunks stored and indexed in Cortex storage backends.
- logtool: Tool which parses Cortex query-frontend logs and formats them for easy analysis.
This tool is designed to interact with the various user-facing APIs provided by Cortex, as well as, interact with various backend storage components containing Cortex data.
Config commands interact with the Cortex api and read/create/update/delete user configs from Cortex. Specifically a users alertmanager and rule configs can be composed and updated using these commands.
| Env Variables | Flag | Description |
|---|---|---|
| CORTEX_ADDRESS | address |
Addess of the API of the desired Cortex cluster. |
| CORTEX_API_KEY | key |
In cases where the Cortex API is set behind a basic auth gateway, an key can be set as a basic auth password. |
| CORTEX_TENANT_ID | id |
The tenant ID of the Cortex instance to interact with. |
The following commands are used by users to interact with their Cortex alertmanager configuration, as well as their alert template files.
cortextool alertmanager get
cortextool alertmanager load ./example_alertmanager_config.yaml
The following commands are used by users to interact with their Cortex ruler configuration. They can load prometheus rule files, as well as interact with individual rule groups.
This command will retrieve all of the rule groups stored in the specified Cortex instance and print each one by rule group name and namespace to the terminal.
cortextool rules list
This command will retrieve all of the rule groups stored in the specified Cortex instance and print them to the terminal.
cortextool rules print
This command will retrieve the specified rule group from Cortex and print it to the terminal.
cortextool rules get example_namespace example_rule_group
This command will retrieve the specified rule group from Cortex and print it to the terminal.
cortextool rules delete example_namespace example_rule_group
This command will load each rule group in the specified files and load them into Cortex. If a rule already exists in Cortex it will be overwritten if a diff is found.
cortextool rules load ./example_rules_one.yaml ./example_rules_two.yaml ...
This command lints a rules file. The linter's aim is not to verify correctness but just YAML and PromQL expression formatting within the rule file. This command always edits in place, you can use the dry run flag (-n) if you'd like to perform a trial run that does not make any changes.
cortextool rules lint -n ./example_rules_one.yaml ./example_rules_two.yaml ...
This command prepares a rules file for upload to Cortex. It lints all your PromQL expressions and adds an specific label to your PromQL query aggregations in the file. Unlike, the previous command this one does not interact with your Cortex cluster.
cortextool rules prepare -i ./example_rules_one.yaml ./example_rules_two.yaml ...
There are two flags of note for this command:
-iwhich allows you to edit in place, otherwise a a new file with a.outputextension is created with the results of the run.-lwhich allows you specify the label you want you add for your aggregations, it isclusterby default.
At the end of the run, the command tells you whenever the operation was a success in the form of
INFO[0000] SUCESS: 194 rules found, 0 modified expressions
It is important to note that a modification can be a PromQL expression lint or a label add to your aggregation.
This commands checks rules against the recommended best practices for rules.
cortextool rules check ./example_rules_one.yaml
This repo also contains the chunktool. A client meant to interact with chunks stored and indexed in cortex backends.
The delete command currently cleans all index entries pointing to chunks in the specified index. Only bigtable and the v10 schema are currently fully supported. This will not delete the entire index entry, only the corresponding chunk entries within the index row.
The migrate command helps with migrating chunks across cortex clusters. It also takes care of setting right index in the new cluster as per the specified schema config.
As of now it only supports Bigtable or GCS as a source to read chunks from for migration while for writing it supports all the storages that Cortex supports.
More details about it here
A CLI tool to parse Cortex query-frontend logs and formats them for easy analysis.
Options:
-dur duration
only show queries which took longer than this duration, e.g. -dur 10s
-query
show the query
-utc
show timestamp in UTC time
Feed logs into it using logcli from Loki, kubectl for Kubernetes, cat from a file, or any other way to get raw logs:
Loki logcli example:
$ logcli query '{cluster="us-central1", name="query-frontend", namespace="dev"}' --limit=5000 --since=3h --forward -o raw | ./logtool -dur 5s
https://logs-dev-ops-tools1.grafana.net/loki/api/v1/query_range?direction=FORWARD&end=1591119479093405000&limit=5000&query=%7Bcluster%3D%22us-central1%22%2C+name%3D%22query-frontend%22%2C+namespace%3D%22dev%22%7D&start=1591108679093405000
Common labels: {cluster="us-central1", container_name="query-frontend", job="dev/query-frontend", level="debug", name="query-frontend", namespace="dev", pod_template_hash="7cd4bf469d", stream="stderr"}
Timestamp TraceID Length Duration Status Path
2020-06-02 10:38:40.34205349 -0400 EDT 1f2533b40f7711d3 12h0m0s 21.92465802s (200) /api/prom/api/v1/query_range
2020-06-02 10:40:25.171649132 -0400 EDT 2ac59421db0000d8 168h0m0s 16.378698276s (200) /api/prom/api/v1/query_range
2020-06-02 10:40:29.698167258 -0400 EDT 3fd088d900160ba8 168h0m0s 20.912864541s (200) /api/prom/api/v1/query_range
$ cat query-frontend-logs.log | ./logtool -dur 5s
Timestamp TraceID Length Duration Status Path
2020-05-26 13:51:15.0577354 -0400 EDT 76b9939fd5c78b8f 6h0m0s 10.249149614s (200) /api/prom/api/v1/query_range
2020-05-26 13:52:15.771988849 -0400 EDT 2e7473ab10160630 10h33m0s 7.472855362s (200) /api/prom/api/v1/query_range
2020-05-26 13:53:46.712563497 -0400 EDT 761f3221dcdd85de 10h33m0s 11.874296689s (200) /api/prom/api/v1/query_range
Licensed Apache 2.0, see LICENSE.