Prometheus Operator configuration reference for Lokomotive

Introduction

The Prometheus Operator for Kubernetes provides easy monitoring definitions for Kubernetes services and deployment and management of Prometheus instances.

Prerequisites

A Lokomotive cluster with a PersistentVolume plugin, e.g. OpenEBS or one of the built-in plugins.

Configuration

Prometheus Operator component configuration example:

component "prometheus-operator" {
  namespace = "monitoring"

  grafana {
    admin_password = "foobar"
    secret_env = { # This might contain sensitive information, declare a variable and define this in `lokocfg.vars`.
      "KEY" = "VERY_SECRET"
    }
    ingress {
      host                       = "grafana.mydomain.net"
      class                      = "contour"
      certmanager_cluster_issuer = "letsencrypt-production"
    }
  }

  prometheus {
    metrics_retention = "14d"
    storage_size      = "50GiB"
    node_selector = {
      "kubernetes.io/hostname" = "worker3"
    }
    ingress {
      host                       = "prometheus.mydomain.net"
      class                      = "contour"
      certmanager_cluster_issuer = "letsencrypt-production"
    }
    watch_labeled_service_monitors = true
    watch_labeled_prometheus_rules = true
    external_labels = {
      "cluster" = var.cluster_name
    }
  }

  alertmanager_retention    = "360h"
  alertmanager_external_url = "https://api.example.com/alertmanager"
  alertmanager_config       = file("alertmanager-config.yaml")
  alertmanager_node_selector = {
    "kubernetes.io/hostname" = "worker3"
  }
}

Create alertmanager-config.yaml file if necessary. Visit the alertmanager configuration for more information.

Note: Make sure the whole file is indented two spaces. That is, there are two spaces before the top level block.

  config:
    global:
      resolve_timeout: 5m
    route:
      group_by:
      - job
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'null'
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
    receivers:
    - name: 'null'

NOTE: Ensure the file alertmanager_config.yaml is added to .gitignore to avoid any accidental exposure of sensitive data. Alternatively you can store the alertmanager configuration in lokocfg.vars as below:

#lokocfg.vars
alertmanager_config = <<EOF
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by:
      - job
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'null'
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
    receivers:
    - name: 'null'
EOF

Attribute reference

Table of all the arguments accepted by the component.

Example:

Argument	Description	Default	Type	Required
`namespace`	Namespace to deploy the Prometheus Operator.	`monitoring`	string	false
`grafana.admin_password`	Password for `admin` user in Grafana. If not provided it is auto generated and stored in secret `prometheus-operator-grafana`.	-	string	false
`grafana.secret_env`	Sensitive environment variables passed to Grafana pod and stored as secret. Read more on manipulating `grafana.ini` using env var here.	-	map(string)	false
`grafana.ingress.host`	Ingress URL host to expose Grafana over the internet. NOTE: When running on Packet, a DNS entry pointing at the ingress controller needs to be created.	-	string	true
`grafana.ingress.class`	Ingress class to use for Grafana ingress.	`contour`	string	false
`grafana.ingress.certmanager_cluster_issuer`	`ClusterIssuer` to be used by cert-manager while issuing TLS certificates. Supported values: `letsencrypt-production`, `letsencrypt-staging`.	`letsencrypt-production`	string	false
`prometheus_operator_node_selector`	Node selector to specify nodes where the Prometheus Operator pods should be deployed.	{}	map(string)	false
`prometheus.metrics_retention`	Time duration Prometheus shall retain data for. Must match the regular expression `[0-9]+(ms\|s\|m\|h\|d\|w\|y)` (milliseconds, seconds, minutes, hours, days, weeks and years).	`10d`	string	false
`prometheus.node_selector`	Node selector to specify nodes where the Prometheus pods should be deployed.	{}	map(string)	false
`prometheus.storage_size`	Storage capacity for the Prometheus in bytes. You can express storage as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.	"50Gi"	string	false
`prometheus.watch_labeled_service_monitors`	By default prometheus operator watches only the ServiceMonitor objects in the cluster that are labeled `release: prometheus-operator`. If set to `false` then all the ServiceMonitors will be watched.	`true`	bool	false
`prometheus.watch_labeled_prometheus_rules`	By default prometheus operator watches only the PrometheusRule objects in the cluster that are labeled `release: prometheus-operator` and `app: prometheus-operator`. If set to `false` then all the PrometheusRule will be watched.	`true`	bool	false
`prometheus.external_labels`	This is the Prometheus parameter with the same name. The labels to add to any time series or alerts when communicating with external systems (federation, remote storage, Alertmanager).	-	map(string)	false
`prometheus.ingress.host`	Ingress URL host to expose Prometheus over the internet. NOTE: When running on Packet, a DNS entry pointing at the ingress controller needs to be created.	-	string	true
`prometheus.ingress.class`	Ingress class to use for Prometheus ingress.	`contour`	string	false
`prometheus.ingress.certmanager_cluster_issuer`	`ClusterIssuer` to be used by cert-manager while issuing TLS certificates. Supported values: `letsencrypt-production`, `letsencrypt-staging`.	`letsencrypt-production`	string	false
`alertmanager_retention`	Time duration Alertmanager shall retain data for. Must match the regular expression `[0-9]+(ms\|s\|m\|h)` (milliseconds, seconds, minutes and hours).	`120h`	string	false
`alertmanager_external_url`	The external URL the Alertmanager instances will be available under. This is necessary to generate correct URLs. This is necessary if Alertmanager is not served from root of a DNS name.	""	string	false
`alertmanager_config`	Provide YAML file path to configure Alertmanager. See https://prometheus.io/docs/alerting/configuration/#configuration-file.	`{"global":{"resolve_timeout":"5m"},"route":{"group_by":["job"],"group_wait":"30s","group_interval":"5m","repeat_interval":"12h","receiver":"null","routes":[{"match":{"alertname":"Watchdog"},"receiver":"null"}]},"receivers":[{"name":"null"}]}`	string	false
`alertmanager_node_selector`	Node selector to specify nodes where the AlertManager pods should be deployed.	{}	map(string)	false
`alertmanager_storage_size`	Storage capacity for the Alertmanager in bytes. You can express storage as a fixed-point integer using one of these suffixes: E, P, T, G, M, K. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki.	"50Gi"	string	false
`disable_webhooks`	Disables validation and mutation webhooks. This might be required on older versions of Kubernetes to install successfully.	false	bool	false
`monitor`	Block, which allows to disable scraping of individual Kubernetes components.	-	object	false
`monitor.etcd`	Controls if the default Prometheus instance should scrape etcd metrics.	true	bool	false
`monitor.kube_controller_manager`	Controls if the default Prometheus instance should scrape kube-controller-manager metrics.	true	bool	false
`monitor.kube_scheduler`	Controls if the default Prometheus instance should scrape kube-scheduler metrics.	true	bool	false
`monitor.kube_proxy`	Controls if the default Prometheus instance should scrape kube-proxy metrics.	true	bool	false
`monitor.kubelet`	Controls if the default Prometheus instance should scrape kubelet metrics.	true	bool	false
`coredns`	Block, which allows to customize, how CoreDNS is scraped.	-	object	false
`coredns.selector`	Defines, how CoreDNS pods should be selected for scraping.	{"k8s-app":"coredns","tier":"control-plane"}	map(string)	false
`storage_class`	Storage Class to use for the storage allowed for Prometheus and Alertmanager.	-	string	false

Applying

To apply the Prometheus Operator component:

lokoctl component apply prometheus-operator

Post-installation

To start monitoring your applications running on Kubernetes. Just create a ServiceMonitor object in that namespace which looks like following:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: openebs
  name: openebs
  namespace: openebs
spec:
  endpoints:
  - path: /metrics
    port: exporter
  namespaceSelector:
    matchNames:
    - openebs
  selector:
    matchLabels:
      openebs.io/cas-type: cstor

Change the labels, endpoints, namespaceSelector, selector fields as you need. To learn more about basics of ServiceMonitor read the docs here and the API Reference can be found here.

Deleting

To destroy the component:

lokoctl component delete prometheus-operator --delete-namespace

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prometheus-operator.md

prometheus-operator.md

Prometheus Operator configuration reference for Lokomotive

Contents

Introduction

Prerequisites

Configuration

Attribute reference

Applying

Post-installation

Deleting

Files

prometheus-operator.md

Latest commit

History

prometheus-operator.md

File metadata and controls

Prometheus Operator configuration reference for Lokomotive

Contents

Introduction

Prerequisites

Configuration

Attribute reference

Applying

Post-installation

Deleting