Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitoring: enable alert rules persistence #898

Merged

Conversation

qiffang
Copy link
Contributor

@qiffang qiffang commented Sep 10, 2019

What problem does this PR solve?

Previously, The new alert rules added by monitoring-reloader will be loss if prometheus restart.
It is unreasonable and we update the logic following rules

  • TiDB Version is not changed, the updated rules should be persistence even though prometheus restart

  • TiDB Version is changed, the updated rules will be loss (It is also stored PV).

  • The Cluster is deleted, the updated rules will be loss (It is also stored PV).

It is related the PR

What is changed and how does it work?

Check List

  • helm chart

Tests

  • Upgrade TIDB Cluster but do not change the TIDB version , the updated files will be persisted

  • Upgrade TIDB Version, Updated rules is loss

Code changes

  • Has Helm charts change

Does this PR introduce a user-facing change?:

change tidb-monitor-reloader image to pingcap/tidb-monitor-reloader:v1.0.1

@qiffang qiffang requested a review from tennix September 10, 2019 10:35
@@ -121,6 +125,8 @@ spec:
imagePullPolicy: {{ .Values.monitor.reloader.imagePullPolicy | default "IfNotPresent" }}
command:
- /bin/reload
- --root-store-path=/data
- --sub-store-path={{ .Values.tidb.image }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command-line argument looks odd, can we rename it to make the name and value match?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reloader 想做成和 TIDB 没什么关联独立存在的 prometheus ui reload 工具,所以取了和 TIDB 无关的名字

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is OK. But maybe we can convert the .Values.tidb.image to the expected path in Helm template.

@tennix tennix changed the title Monitoring: support change alert rules persistence Monitoring: enable alert rules persistence Sep 18, 2019
tennix
tennix previously approved these changes Oct 11, 2019
Copy link
Member

@tennix tennix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

weekface
weekface previously approved these changes Oct 11, 2019
Copy link
Contributor

@weekface weekface left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
But, the release-note is not related to this PR fully.

@qiffang qiffang dismissed stale reviews from weekface and tennix via ca0ab71 October 12, 2019 03:11
@qiffang
Copy link
Contributor Author

qiffang commented Oct 12, 2019

/run-e2e-in-kind

@qiffang qiffang added status/PTAL PR needs to be reviewed and removed status/LGTM2 labels Oct 14, 2019
Copy link
Member

@tennix tennix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@weekface
Copy link
Contributor

/run-e2e-in-kind

Copy link
Contributor

@weekface weekface left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qiffang
Copy link
Contributor Author

qiffang commented Oct 18, 2019

/run-e2e-in-kind

@tennix tennix removed the status/PTAL PR needs to be reviewed label Oct 18, 2019
@tennix tennix merged commit b8df97d into pingcap:master Oct 18, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Oct 18, 2019

cherry pick to release-1.0 failed

qiffang added a commit to qiffang/tidb-operator that referenced this pull request Dec 23, 2019
yahonda pushed a commit that referenced this pull request Dec 27, 2021
* update tkctl debug usage

* Update en/use-tkctl.md

* Update en/use-tkctl.md

Co-authored-by: Ran <huangran@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants