Add alertmanager inhibit rules #1925

ijungmann · 2021-05-04T08:34:38Z

Detailed description:
Add inhibit rules to the helm charts to ensure that alerts which could be triggered downstream from another alert are not fired. This will help to keep the alert channel less noisy.

Add a new alert to each service to check when pods go down.
Add inhibit rules to each values.yaml to logically group alerts together
Add new alertmanagerconfig template to handle the new inhibit rules section.
Update the mirror-node-common chart to read the new config

Which issue(s) this PR fixes:
Fixes #1813

Special notes for your reviewer:

Checklist

Documentation added
Tests updated

codecov · 2021-05-04T08:37:53Z

Codecov Report

Merging #1925 (4c132ab) into master (126751e) will increase coverage by 0.68%.
The diff coverage is 79.92%.

❗ Current head 4c132ab differs from pull request most recent head c7e6a36. Consider uploading reports for the commit c7e6a36 to get more accurate results

@@             Coverage Diff              @@
##             master    #1925      +/-   ##
============================================
+ Coverage     87.05%   87.73%   +0.68%     
- Complexity     1744     1822      +78     
============================================
  Files           315      319       +4     
  Lines          7731     8063     +332     
  Branches        740      814      +74     
============================================
+ Hits           6730     7074     +344     
+ Misses          772      744      -28     
- Partials        229      245      +16

Impacted Files	Coverage Δ	Complexity Δ
...ra/mirror/monitor/config/MonitorConfiguration.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
.../hedera/mirror/monitor/publish/PublishMetrics.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...a/mirror/monitor/publish/TransactionPublisher.java	`1.06% <0.00%> (-0.08%)`	`1.00 <0.00> (ø)`
...a/mirror/monitor/subscribe/AbstractSubscriber.java	`77.14% <ø> (-20.00%)`	`3.00 <0.00> (-1.00)`
.../mirror/monitor/subscribe/rest/RestSubscriber.java	`100.00% <ø> (ø)`	`22.00 <0.00> (?)`
...nitor/subscribe/rest/RestSubscriberProperties.java	`83.33% <ø> (ø)`	`4.00 <0.00> (?)`
hedera-mirror-rosetta/cmd/db.go	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ra/mirror/monitor/subscribe/SubscribeResponse.java	`66.66% <66.66%> (ø)`	`4.00 <4.00> (?)`
hedera-mirror-rosetta/cmd/config.go	`66.66% <67.74%> (+66.66%)`	`0.00 <0.00> (ø)`
...irror/monitor/subscribe/grpc/GrpcSubscription.java	`70.00% <70.00%> (ø)`	`15.00 <15.00> (?)`
... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 126751e...c7e6a36. Read the comment docs.

charts/hedera-mirror-importer/values.yaml

charts/hedera-mirror-grpc/values.yaml

charts/hedera-mirror-grpc/templates/alertmanagerconfig.yaml

charts/hedera-mirror-grpc/values.yaml

charts/hedera-mirror-importer/values.yaml

xin-hedera

looks good, just some minor suggested changes

xin-hedera · 2021-05-11T16:00:34Z

charts/hedera-mirror-grpc/templates/alertmanagerconfig.yaml

+  labels:
+    {{ include "hedera-mirror-grpc.labels" . | nindent 4 }}


Suggested change

labels:

{{ include "hedera-mirror-grpc.labels" . | nindent 4 }}

labels: {{ include "hedera-mirror-grpc.labels" . | nindent 4 }}

Fixed in all templates.

xin-hedera · 2021-05-11T16:04:31Z

charts/hedera-mirror-grpc/templates/alertmanagerconfig.yaml

+  inhibitRules:
+    {{- range $name, $rule := omit .Values.alertmanager.inhibitRules "enabled" }}
+      {{- if $rule.enabled }}
+        {{- $rule.matches | toYaml | nindent 4}}


Suggested change

{{- $rule.matches | toYaml | nindent 4}}

{{- $rule.matches | toYaml | nindent 4 }}

Fixed in all templates.

xin-hedera · 2021-05-11T16:05:50Z

charts/hedera-mirror-importer/values.yaml

+          equal:
+            - namespace
+            - pod
+            -


Suggested change

-

charts/hedera-mirror-importer/values.yaml

ijungmann · 2021-05-11T16:33:38Z

charts/hedera-mirror-rest/values.yaml

@@ -141,11 +173,11 @@ readinessProbe:

 resources:
  limits:
-    cpu: 500m
-    memory: 128Mi
+    cpu: 1.8


Yep, had these swapped with the importer resources. Fixed now.

ijungmann · 2021-05-11T16:33:49Z

charts/hedera-mirror/values-prod.yaml

@@ -3,20 +3,26 @@ grpc:
  ingress:
    middleware:
      enabled: true
+  inhibitRules:


charts/hedera-mirror-grpc/values.yaml

charts/hedera-mirror-monitor/values.yaml

charts/hedera-mirror-grpc/values.yaml

charts/hedera-mirror-importer/values.yaml

charts/hedera-mirror/values.yaml

Nana-EC · 2021-05-13T23:53:38Z

charts/hedera-mirror-importer/values.yaml

+        - sourceMatch:
+            - name: alertname
+              regex: true
+              value: (ImporterParseErrors)


nit: no need for a regex since it's just one value

charts/hedera-mirror-grpc/values.yaml

charts/hedera-mirror-importer/values.yaml

ijungmann · 2021-05-14T07:12:35Z

charts/hedera-mirror-importer/values.yaml

@@ -110,6 +202,10 @@ prometheusRules:
    for: 1m
    labels:
      severity: critical
+      application: hedera-mirror-importer
+      type: BALANCE


Added type to a few of the rules where it can technically be inferred, but needs to be present to group in the inhibit rules without defining a separate rule.

charts/hedera-mirror-monitor/values.yaml

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann · 2021-05-17T16:23:28Z

Fixed DCO issue

sonarcloud · 2021-05-17T16:27:46Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
15 Code Smells

0.0% Coverage
0.3% Duplication

* Add a new alert to each service to check when pods are not ready * Add inhibit rules to each values.yaml to logically group alerts together * Add new alertmanagerconfig template to handle the new inhibit rules section * Update the mirror-node-common chart to read the new config Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

steven-sheehy added this to the Mirror 0.34.0 milestone May 5, 2021

ijungmann added enhancement Type: New feature ops Tasks relating to network operations P2 labels May 5, 2021

ijungmann self-assigned this May 5, 2021

ijungmann force-pushed the add_alert_inhibit_rules branch 2 times, most recently from 9c55a4b to c6ed080 Compare May 6, 2021 06:53

ijungmann marked this pull request as ready for review May 6, 2021 06:57

ijungmann requested a review from a team May 6, 2021 06:57

ijungmann commented May 6, 2021

View reviewed changes

charts/hedera-mirror-importer/values.yaml Outdated Show resolved Hide resolved

ijungmann commented May 6, 2021

View reviewed changes

charts/hedera-mirror-grpc/values.yaml Outdated Show resolved Hide resolved

ijungmann commented May 6, 2021

View reviewed changes

charts/hedera-mirror-grpc/values.yaml Outdated Show resolved Hide resolved

ijungmann commented May 6, 2021

View reviewed changes

charts/hedera-mirror-grpc/values.yaml Outdated Show resolved Hide resolved

steven-sheehy requested changes May 7, 2021

View reviewed changes

ijungmann commented May 10, 2021

View reviewed changes

charts/hedera-mirror-importer/values.yaml Outdated Show resolved Hide resolved

ijungmann force-pushed the add_alert_inhibit_rules branch from 477f95e to 6951127 Compare May 10, 2021 05:21

ijungmann requested a review from steven-sheehy May 11, 2021 00:21

xin-hedera requested changes May 11, 2021

View reviewed changes

ijungmann commented May 11, 2021

View reviewed changes

steven-sheehy requested changes May 11, 2021

View reviewed changes

steven-sheehy requested changes May 12, 2021

View reviewed changes

steven-sheehy reviewed May 13, 2021

View reviewed changes

charts/hedera-mirror/values.yaml Show resolved Hide resolved

Nana-EC reviewed May 14, 2021

View reviewed changes

ijungmann requested a review from xin-hedera May 14, 2021 06:34

ijungmann commented May 14, 2021

View reviewed changes

charts/hedera-mirror-grpc/values.yaml Show resolved Hide resolved

ijungmann commented May 14, 2021

View reviewed changes

charts/hedera-mirror-importer/values.yaml Outdated Show resolved Hide resolved

ijungmann commented May 14, 2021

View reviewed changes

charts/hedera-mirror-monitor/values.yaml Outdated Show resolved Hide resolved

Ian Jungmann added 19 commits May 17, 2021 11:22

Fix pod name rule

2659d44

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Move inhibitRules under alertmanager in prod, and fix resources

a35d9c2

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Template fixes, cleanup

a16cee2

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

PR comments

8ad4e77

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Add top level inhibit rules

1997378

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Cleanup

945a1b4

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Remove useless rule

069bede

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Cleanup

7c2d09a

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Remove useless rule

bfb4ee7

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Add template

178be40

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Cleanup cloud rules

9a26541

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Really clean up importer rules

5dcd455

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Really clean up importer rules

e305574

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Cleanup

8cc6289

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

More labels, more cleanup

648462a

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Rename rule

37bfd70

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Swap some labels out for regex

55804ea

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Bad rule

5291b86

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

Fix indentation

c7e6a36

Signed-off-by: Ian Jungmann <ian.jungmann@hedera.com>

ijungmann dismissed stale reviews from steven-sheehy and Nana-EC via c7e6a36 May 17, 2021 16:23

ijungmann force-pushed the add_alert_inhibit_rules branch from 7ddb2f6 to c7e6a36 Compare May 17, 2021 16:23

xin-hedera approved these changes May 17, 2021

View reviewed changes

steven-sheehy approved these changes May 17, 2021

View reviewed changes

ijungmann merged commit 229d112 into master May 17, 2021

ijungmann deleted the add_alert_inhibit_rules branch May 17, 2021 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add alertmanager inhibit rules #1925

Add alertmanager inhibit rules #1925

ijungmann commented May 4, 2021 •

edited

Loading

codecov bot commented May 4, 2021 •

edited

Loading

xin-hedera left a comment

xin-hedera May 11, 2021

ijungmann May 11, 2021

xin-hedera May 11, 2021

ijungmann May 11, 2021

xin-hedera May 11, 2021

ijungmann May 11, 2021

ijungmann May 11, 2021

ijungmann May 11, 2021

Nana-EC May 13, 2021

ijungmann May 14, 2021

ijungmann May 14, 2021

ijungmann commented May 17, 2021

sonarcloud bot commented May 17, 2021

		labels:
		{{ include "hedera-mirror-grpc.labels" . \| nindent 4 }}

	labels:
	{{ include "hedera-mirror-grpc.labels" . \| nindent 4 }}
	labels: {{ include "hedera-mirror-grpc.labels" . \| nindent 4 }}

	{{- $rule.matches \| toYaml \| nindent 4}}
	{{- $rule.matches \| toYaml \| nindent 4 }}

Add alertmanager inhibit rules #1925

Add alertmanager inhibit rules #1925

Conversation

ijungmann commented May 4, 2021 • edited Loading

codecov bot commented May 4, 2021 • edited Loading

Codecov Report

xin-hedera left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ijungmann commented May 17, 2021

sonarcloud bot commented May 17, 2021

ijungmann commented May 4, 2021 •

edited

Loading

codecov bot commented May 4, 2021 •

edited

Loading