Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use original file in rules loaded metric #1857

Closed
wants to merge 1 commit into from

Conversation

yeya24
Copy link
Contributor

@yeya24 yeya24 commented Dec 10, 2019

Signed-off-by: yeya24 yb532204897@gmail.com

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Use the original rule file name as the label in load rules metric

Verification

Signed-off-by: yeya24 <yb532204897@gmail.com>
@yeya24
Copy link
Contributor Author

yeya24 commented Dec 10, 2019

There are still some wrong metrics exported by rule manager, but now I don't figure out how to fix them, it may change a lot

curl http://localhost:10908/metrics | grep data/.tmp

prometheus_rule_group_interval_seconds{rule_group="data/.tmp-rules/valid.yaml.96e6e93b5a37e1df396d2d69d8cdba3ec79dd212157123e6bcc2045f28033275.WARN;test-alert-group",strategy="warn"} 5
prometheus_rule_group_interval_seconds{rule_group="data/.tmp-rules/valid.yaml.96e6e93b5a37e1df396d2d69d8cdba3ec79dd212157123e6bcc2045f28033275.WARN;test-alert-group1",strategy="warn"} 5
prometheus_rule_group_last_duration_seconds{rule_group="data/.tmp-rules/valid.yaml.96e6e93b5a37e1df396d2d69d8cdba3ec79dd212157123e6bcc2045f28033275.WARN;test-alert-group",strategy="warn"} 0.001195467
prometheus_rule_group_last_duration_seconds{rule_group="data/.tmp-rules/valid.yaml.96e6e93b5a37e1df396d2d69d8cdba3ec79dd212157123e6bcc2045f28033275.WARN;test-alert-group1",strategy="warn"} 0.002537008
prometheus_rule_group_last_evaluation_timestamp_seconds{rule_group="data/.tmp-rules/valid.yaml.96e6e93b5a37e1df396d2d69d8cdba3ec79dd212157123e6bcc2045f28033275.WARN;test-alert-group",strategy="warn"} 1.575936400578541e+09
prometheus_rule_group_last_evaluation_timestamp_seconds{rule_group="data/.tmp-rules/valid.yaml.96e6e93b5a37e1df396d2d69d8cdba3ec79dd212157123e6bcc2045f28033275.WARN;test-alert-group1",strategy="warn"} 1.575936402443577e+09
prometheus_rule_group_rules{rule_group="data/.tmp-rules/valid.yaml.96e6e93b5a37e1df396d2d69d8cdba3ec79dd212157123e6bcc2045f28033275.WARN;test-alert-group",strategy="warn"} 1
prometheus_rule_group_rules{rule_group="data/.tmp-rules/valid.yaml.96e6e93b5a37e1df396d2d69d8cdba3ec79dd212157123e6bcc2045f28033275.WARN;test-alert-group1",strategy="warn"} 1

@bwplotka
Copy link
Member

Thanks for noticing this. It's not perfect.

What I will propose is to actually propose PartialResponseStrategy on Rules and Alerts in the upstream Prometheus. This might be as well needed as even Prometheus has something like res.Warnings in places like rule/alert evaluation: https://github.com/prometheus/prometheus/blob/06066a3619f0561a6e048b8c7cb874985c5a443b/rules/manager.go#L169

This will make native rule/alert being compatible with Thanos one, thus solving this. (:

cc @simonpasquier if that makes sesne (:

@simonpasquier
Copy link
Contributor

@bwplotka interesting idea indeed!

@stale
Copy link

stale bot commented Jan 11, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jan 11, 2020
@stale stale bot closed this Jan 18, 2020
@bwplotka bwplotka reopened this Jan 20, 2020
@stale stale bot removed the stale label Jan 20, 2020
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's stale but also it's really quite bad -> it introduces some unnecessary cardinality. Let's fix it!

LGTM 👍

@stale
Copy link

stale bot commented Feb 19, 2020

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

@stale stale bot added stale and removed stale labels Feb 19, 2020
@yeya24
Copy link
Contributor Author

yeya24 commented Feb 21, 2020

This PR is still valid.

@stale
Copy link

stale bot commented Mar 22, 2020

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

@stale stale bot added the stale label Mar 22, 2020
@stale stale bot closed this Mar 29, 2020
@yeya24
Copy link
Contributor Author

yeya24 commented Mar 29, 2020

Sorry about that. Just notice I need a rebase in this PR... Is it possible to make this open?

@bwplotka
Copy link
Member

bwplotka commented Mar 29, 2020 via email

@yeya24
Copy link
Contributor Author

yeya24 commented Mar 29, 2020

I don't have the permission to re-open it.

@bwplotka
Copy link
Member

I cannot as well and this is why:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants