Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add alertmanager provider #258

Merged
merged 1 commit into from
Oct 19, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion api/v1beta1/provider_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ const (
// ProviderSpec defines the desired state of Provider
type ProviderSpec struct {
// Type of provider
// +kubebuilder:validation:Enum=slack;discord;msteams;rocket;generic;github;gitlab;bitbucket;azuredevops;googlechat;webex;sentry;azureeventhub;telegram;lark;matrix;opsgenie;
// +kubebuilder:validation:Enum=slack;discord;msteams;rocket;generic;github;gitlab;bitbucket;azuredevops;googlechat;webex;sentry;azureeventhub;telegram;lark;matrix;opsgenie;alertmanager;
// +required
Type string `json:"type"`

Expand Down Expand Up @@ -81,6 +81,7 @@ const (
LarkProvider string = "lark"
Matrix string = "matrix"
OpsgenieProvider string = "opsgenie"
AlertManagerProvider string = "alertmanager"
)

// ProviderStatus defines the observed state of Provider
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ spec:
- lark
- matrix
- opsgenie
- alertmanager
type: string
username:
description: Bot username for this provider
Expand Down
37 changes: 35 additions & 2 deletions docs/spec/v1beta1/provider.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Notification providers:
* Azure Event Hub
* Generic webhook
* Opsgenie
* Alertmanager

Git commit status providers:

Expand Down Expand Up @@ -117,7 +118,7 @@ kubectl create secret generic webhook-url \
Note that the secret must contain an `address` field.

The provider type can be: `slack`, `msteams`, `rocket`, `discord`, `googlechat`, `webex`, `sentry`,
`telegram`, `lark`, `matrix`, `azureeventhub` or `generic`.
`telegram`, `lark`, `matrix`, `azureeventhub`, `opsgenie`, `alertmanager` or `generic`.

When type `generic` is specified, the notification controller will post the
incoming [event](event.md) in JSON format to the webhook address.
Expand Down Expand Up @@ -279,6 +280,39 @@ spec:
```


### Prometheus Alertmanager

Sends notifications to [alertmanager v2 api](https://github.com/prometheus/alertmanager/blob/main/api/v2/openapi.yaml) if alert manager has basic authentication configured it is recommended to use
secretRef and include the username:password in the address string.

```yaml
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
name: alertmanager
namespace: default
spec:
type: alertmanager
# webhook address (ignored if secretRef is specified)
address: https://....@<alertmanager-url>/api/v2/alerts/"
```

When an event is triggered the provider will send a single alert with at least one annotation for alert which is the "message" found for the event.
If a summary is provided in the alert resource an additional "summary" annotation will be added.

Alan01252 marked this conversation as resolved.
Show resolved Hide resolved
The provider will send the following labels for the event.


| Label | Description |
| ----------- | -------------------------------------------------------------------------------------------------- |
| alertname | The string Flux followed by the Kind and the reason for the event e.g FluxKustomizationProgressing |
| severity | The severity of the event (error|info) |
| reason | The machine readable reason for the objects transition into the current status |
| kind | The kind of the involved object associated with the event |
| name | The name of the involved object associated with the event |
| namespace | The namespace of the involved object associated with the event |


### Git commit status

The GitHub, GitLab, Bitbucket, and Azure DevOps provider will write to the
Expand Down Expand Up @@ -340,7 +374,6 @@ data:
token: <username>:<app-password>
```


Opsgenie uses an api key to authenticate [api key](https://support.atlassian.com/opsgenie/docs/api-key-management/).
The providers require a secret in the same format, with the api key as the value for the token key:

Expand Down
92 changes: 92 additions & 0 deletions internal/notifier/alertmanager.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
/*
Copyright 2020 The Flux authors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package notifier

import (
"crypto/x509"
"fmt"
"net/url"
"strings"

"github.com/fluxcd/pkg/runtime/events"
)

type Alertmanager struct {
URL string
ProxyURL string
CertPool *x509.CertPool
}

type AlertManagerAlert struct {
Status string `json:"status"`
Labels map[string]string `json:"labels"`
Annotations map[string]string `json:"annotations"`
}

func NewAlertmanager(hookURL string, proxyURL string, certPool *x509.CertPool) (*Alertmanager, error) {
_, err := url.ParseRequestURI(hookURL)
if err != nil {
return nil, fmt.Errorf("invalid Alertmanager URL %s", hookURL)
}

return &Alertmanager{
URL: hookURL,
ProxyURL: proxyURL,
CertPool: certPool,
}, nil
}

func (s *Alertmanager) Post(event events.Event) error {
// Skip any update events
if isCommitStatus(event.Metadata, "update") {
return nil
}

annotations := make(map[string]string)
annotations["message"] = event.Message

_, ok := event.Metadata["summary"]
if ok {
annotations["summary"] = event.Metadata["summary"]
delete(event.Metadata, "summary")
}

labels := event.Metadata
labels["alertname"] = "Flux" + event.InvolvedObject.Kind + strings.Title(event.Reason)
labels["severity"] = event.Severity
labels["reason"] = event.Reason

labels["kind"] = event.InvolvedObject.Kind
labels["name"] = event.InvolvedObject.Name
labels["namespace"] = event.InvolvedObject.Namespace
labels["reportingcontroller"] = event.ReportingController

payload := []AlertManagerAlert{
{
Labels: labels,
Annotations: annotations,
Status: "firing",
},
}

err := postMessage(s.URL, s.ProxyURL, s.CertPool, payload)

if err != nil {
return fmt.Errorf("postMessage failed: %w", err)
}
return nil
}
45 changes: 45 additions & 0 deletions internal/notifier/alertmanger_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/*
Copyright 2020 The Flux authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package notifier

import (
"encoding/json"
"io/ioutil"
"net/http"
"net/http/httptest"
"testing"

"github.com/stretchr/testify/require"
)

func TestAlertmanager_Post(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
b, err := ioutil.ReadAll(r.Body)
require.NoError(t, err)
var payload []AlertManagerAlert
err = json.Unmarshal(b, &payload)
require.NoError(t, err)

}))
defer ts.Close()

alertmanager, err := NewAlertmanager(ts.URL, "", nil)
require.NoError(t, err)

err = alertmanager.Post(testEvent())
require.NoError(t, err)
}
2 changes: 2 additions & 0 deletions internal/notifier/factory.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,8 @@ func (f Factory) Notifier(provider string) (Interface, error) {
n, err = NewMatrix(f.URL, f.Token, f.Channel)
case v1beta1.OpsgenieProvider:
n, err = NewOpsgenie(f.URL, f.ProxyURL, f.CertPool, f.Token)
case v1beta1.AlertManagerProvider:
n, err = NewAlertmanager(f.URL, f.ProxyURL, f.CertPool)
default:
err = fmt.Errorf("provider %s not supported", provider)
}
Expand Down
1 change: 0 additions & 1 deletion internal/notifier/opsgenie.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ type OpsgenieAlert struct {
Details map[string]string `json:"details"`
}

// NewSlack validates the Slack URL and returns a Slack object
func NewOpsgenie(hookURL string, proxyURL string, certPool *x509.CertPool, token string) (*Opsgenie, error) {
_, err := url.ParseRequestURI(hookURL)
if err != nil {
Expand Down