From c169148043c8916b98c0c14e6fee32cfd754f824 Mon Sep 17 00:00:00 2001 From: Ran Date: Fri, 8 Jan 2021 11:40:27 +0800 Subject: [PATCH] en: refactor monitoring docs (#1020) * en: refactor monitoring docs Signed-off-by: Ran * add aliases Signed-off-by: Ran * fix deadlink Signed-off-by: Ran * Apply suggestions from code review Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- en/TOC.md | 4 +- en/access-dashboard.md | 2 +- en/enable-tidb-cluster-auto-scaling.md | 2 +- en/get-started.md | 2 +- en/monitor-a-tidb-cluster.md | 286 +++++++++++++++++++---- en/monitor-kubernetes.md | 47 ++++ en/monitor-using-tidbmonitor.md | 301 ------------------------- en/notes-tidb-operator-v1.1.md | 2 +- zh/monitor-a-tidb-cluster.md | 8 +- 9 files changed, 294 insertions(+), 360 deletions(-) create mode 100644 en/monitor-kubernetes.md delete mode 100644 en/monitor-using-tidbmonitor.md diff --git a/en/TOC.md b/en/TOC.md index fd54db66b5..c49e58ec65 100644 --- a/en/TOC.md +++ b/en/TOC.md @@ -29,8 +29,7 @@ - [Deploy TiDB Binlog](deploy-tidb-binlog.md) - [Deploy TiDB Enterprise Edition](deploy-tidb-enterprise-edition.md) + Deploy Monitoring - - [Monitor Kubernetes and TiDB Cluster](monitor-a-tidb-cluster.md) - - [Monitor TiDB Cluster Using TidbMonitor](monitor-using-tidbmonitor.md) + - [Deploy Monitoring and Alerts for TiDB](monitor-a-tidb-cluster.md) - [Access TiDB Dashboard](access-dashboard.md) + Secure - [Enable TLS for the MySQL Client](enable-tls-for-mysql-client.md) @@ -90,6 +89,7 @@ - [Configure tidb-cluster Chart](tidb-cluster-chart-config.md) - [Configure tidb-backup Chart](configure-backup.md) - [Log Collection](logs-collection.md) + - [Monitoring and Alert on Kubernetes](monitor-kubernetes.md) + [TiDB Operator Roadmap](roadmap.md) + Release Notes + v1.1 diff --git a/en/access-dashboard.md b/en/access-dashboard.md index 15e1d8fcea..d4508fafaa 100644 --- a/en/access-dashboard.md +++ b/en/access-dashboard.md @@ -208,4 +208,4 @@ Due to the special environment of Kubernetes, some features of TiDB Dashboard ar - The log search feature is unavailable. If you need to view the log of a component, execute `kubectl logs ${pod_name} -n {namespace}`. You can also view logs using the log service of the Kubernetes cluster. -- In **Cluster Info** -> **Hosts**, the **Disk Usage** cannot display correctly. You can view the disk usage of each component by viewing the component dashboards in [the TidbMonitor dashboard](monitor-a-tidb-cluster.md#access-the-grafana-monitoring-dashboard). You can also view the disk usage of Kubernetes nodes by deploying a [Kubernetes host monitoring system](monitor-a-tidb-cluster.md#monitor-kubernetes-components). +- In **Cluster Info** -> **Hosts**, the **Disk Usage** cannot display correctly. You can view the disk usage of each component by viewing the component dashboards in [the TidbMonitor dashboard](monitor-a-tidb-cluster.md#access-the-grafana-monitoring-dashboard). You can also view the disk usage of Kubernetes nodes by deploying a [Kubernetes host monitoring system](monitor-kubernetes.md#monitor-kubernetes-components). diff --git a/en/enable-tidb-cluster-auto-scaling.md b/en/enable-tidb-cluster-auto-scaling.md index b01d861f7d..54d47acbb3 100644 --- a/en/enable-tidb-cluster-auto-scaling.md +++ b/en/enable-tidb-cluster-auto-scaling.md @@ -85,7 +85,7 @@ spec: The TiDB component can be configured using `spec.tidb`. Currently, the auto-scaling API of TiDB is the same as that of TiKV. -In a `TidbClusterAutoScaler` object, the `cluster` attribute specifies the TiDB clusters to be auto-scaled. These clusters are marked by `name` and `namespace`. You need to provide the metrics collection and query service to `TidbClusterAutoScaler` because it captures resource usage through the metrics collection component. The `monitor` attribute refers to the `TidbMonitor` object. For more information, see [Monitor TiDB Clusters using TidbMonitor](monitor-using-tidbmonitor.md). +In a `TidbClusterAutoScaler` object, the `cluster` attribute specifies the TiDB clusters to be auto-scaled. These clusters are marked by `name` and `namespace`. You need to provide the metrics collection and query service to `TidbClusterAutoScaler` because it captures resource usage through the metrics collection component. The `monitor` attribute refers to the `TidbMonitor` object. For more information, see [Deploy Monitoring and Alerts for a TiDB Cluster](monitor-a-tidb-cluster.md). For the external `Prometheus` other than `TidbMonitor`, you can fill in the Host by configuring `spec.metricsUrl` to specify the monitoring metrics collection service for the TiDB cluster. If you deploy the monitoring of the TiDB cluster using `Helm`, take the following steps to specify `spec.metricsUrl`. diff --git a/en/get-started.md b/en/get-started.md index 0e501fd32f..d66e8c4158 100644 --- a/en/get-started.md +++ b/en/get-started.md @@ -619,7 +619,7 @@ You can access Grafana dashboard at on the host where yo The default username and password in Grafana are both `admin`. -For more information about monitoring the TiDB cluster in TiDB Operator, refer to [Monitor a TiDB Cluster Using TidbMonitor](monitor-using-tidbmonitor.md). +For more information about monitoring the TiDB cluster in TiDB Operator, refer to [Deploy Monitoring and Alerts for a TiDB Cluster](monitor-a-tidb-cluster.md). ## Upgrade a TiDB cluster diff --git a/en/monitor-a-tidb-cluster.md b/en/monitor-a-tidb-cluster.md index 26514a4531..8dc51da88d 100644 --- a/en/monitor-a-tidb-cluster.md +++ b/en/monitor-a-tidb-cluster.md @@ -1,24 +1,105 @@ --- -title: Monitor and Alert of Kubernetes and the TiDB Cluster +title: Deploy Monitoring and Alerts for a TiDB Cluster summary: Learn how to monitor a TiDB cluster in Kubernetes. -aliases: ['/docs/tidb-in-kubernetes/dev/monitor-a-tidb-cluster/'] +aliases: ['/docs/tidb-in-kubernetes/dev/monitor-a-tidb-cluster/','/docs/tidb-in-kubernetes/dev/monitor-using-tidbmonitor/','/tidb-in-kubernetes/dev/monitor-using-tidbmonitor/'] --- -# Monitor and Alert of Kubernetes and the TiDB Cluster +# Deploy Monitoring and Alerts for a TiDB Cluster -Monitoring a TiDB cluster deployed in Kubernetes can be roughly divided into two parts: - -- Monitoring **the TiDB cluster itself** -- Monitoring **the Kubernetes cluster** and **TiDB Operator** +This document describes how to monitor a TiDB cluster deployed using TiDB Operator and configure alerts for the cluster. ## Monitor the TiDB cluster -You can monitor the TiDB cluster with Prometheus and Grafana. When you create a new TiDB cluster using TiDB Operator, refer to [TidbMonitor Configuration](monitor-using-tidbmonitor.md) to deploy a separate monitoring system for the TiDB cluster. The monitoring system must run in the same namespace as the TiDB cluster, and includes two components: Prometheus and Grafana. - -The monitoring data is not persisted by default. To persist the monitoring data, you can set `spec.persistent` to `true` in `TidbMonitor`. When you enable this option, you need to set `spec.storageClassName` to an existing storage in the current cluster, and this storage is required to support persisting data; otherwise, there is a risk of data loss. +You can monitor the TiDB cluster with Prometheus and Grafana. When you create a new TiDB cluster using TiDB Operator, you can deploy a separate monitoring system for the TiDB cluster. The monitoring system must run in the same namespace as the TiDB cluster, and includes two components: Prometheus and Grafana. For configuration details on the monitoring system, refer to [TiDB Cluster Monitoring](https://pingcap.com/docs/stable/how-to/monitor/monitor-a-cluster). +In TiDB Operator v1.1 or later versions, you can monitor a TiDB cluster on a Kubernetes cluster by using a simple Custom Resource (CR) file called `TidbMonitor`. + +> **Note:** +> +> * One `TidbMonitor` can only monitor one `TidbCluster`. +> * `spec.clusters[0].name` should be set to the `TidbCluster` name of the corresponding TiDB cluster. + +### Persist monitoring data + +The monitoring data is not persisted by default. To persist the monitoring data, you can set `spec.persistent` to `true` in `TidbMonitor`. When you enable this option, you need to set `spec.storageClassName` to an existing storage in the current cluster. This storage must support persisting data; otherwise, there is a risk of data loss. + +A configuration example is as follows: + +```yaml +apiVersion: pingcap.com/v1alpha1 +kind: TidbMonitor +metadata: + name: basic +spec: + clusters: + - name: basic + persistent: true + storageClassName: ${storageClassName} + storage: 5G + prometheus: + baseImage: prom/prometheus + version: v2.18.1 + service: + type: NodePort + grafana: + baseImage: grafana/grafana + version: 6.1.6 + service: + type: NodePort + initializer: + baseImage: pingcap/tidb-monitor-initializer + version: v4.0.9 + reloader: + baseImage: pingcap/tidb-monitor-reloader + version: v1.0.1 + imagePullPolicy: IfNotPresent +``` + +To verify the PVC status, run the following command: + +{{< copyable "shell-regular" >}} + +```shell +kubectl get pvc -l app.kubernetes.io/instance=basic,app.kubernetes.io/component=monitor -n ${namespace} +``` + +``` +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +basic-monitor Bound pvc-6db79253-cc9e-4730-bbba-ba987c29db6f 5G RWO standard 51s +``` + +### Customize the Prometheus configuration + +You can customize the Prometheus configuration by using a customized configuration file or by adding extra options to the command. + +#### Use a customized configuration file + +1. Create a ConfigMap for your customized configuration, and set the key name of `data` to `prometheus-config`. +2. Set `spec.prometheus.config.configMapRef.name` and `spec.prometheus.config.configMapRef.namespace` to the name and namespace of the customized ConfigMap respectively. + +For the complete configuration, refer to the [tidb-operator example](https://github.com/pingcap/tidb-operator/blob/master/examples/monitor-with-externalConfigMap/README.md). + +#### Add extra options to the command + +To add extra options to the command that starts Prometheus, configure `spec.prometheus.config.commandOptions`. + +For the complete configuration, refer to the [tidb-operator example](https://github.com/pingcap/tidb-operator/blob/master/examples/monitor-with-externalConfigMap/README.md). + +> **Note:** +> +> The following options are automatically configured by the TidbMonitor controller and cannot be specified again via `commandOptions`. +> +> - `config.file` +> - `log.level` +> - `web.enable-admin-api` +> - `web.enable-lifecycle` +> - `storage.tsdb.path` +> - `storage.tsdb.retention` +> - `storage.tsdb.max-block-duration` +> - `storage.tsdb.min-block-duration` + ### Access the Grafana monitoring dashboard You can run the `kubectl port-forward` command to access the Grafana monitoring dashboard: @@ -31,7 +112,7 @@ kubectl port-forward -n ${namespace} svc/${cluster_name}-grafana 3000:3000 &>/tm Then open [http://localhost:3000](http://localhost:3000) in your browser and log on with the default username and password `admin`. -You can also set `spec.grafana.service.type` to `NodePort` or `LoadBalancer`, and then view the monitoring dashboard through `NodePort` or `LoadBalancer`. For details, see [TidbMonitor Configuration](monitor-using-tidbmonitor.md). +You can also set `spec.grafana.service.type` to `NodePort` or `LoadBalancer`, and then view the monitoring dashboard through `NodePort` or `LoadBalancer`. If there is no need to use Grafana, you can delete the part of `spec.grafana` in `TidbMonitor` during deployment. In this case, you need to use other existing or newly deployed data visualization tools to directly access the monitoring data. @@ -47,60 +128,165 @@ kubectl port-forward -n ${namespace} svc/${cluster_name}-prometheus 9090:9090 &> Then open [http://localhost:9090](http://localhost:9090) in your browser or access this address via a client tool. -You can also set `spec.prometheus.service.type` to `NodePort` or `LoadBalancer`, and then view the monitoring data through `NodePort` or `LoadBalancer`. For details, see [TidbMonitor Configuration](monitor-using-tidbmonitor.md). - -## Monitor the Kubernetes cluster - -The TiDB monitoring system deployed with the cluster only focuses on the operation of the TiDB components themselves, and does not include the monitoring of container resources, hosts, Kubernetes components, and TiDB Operator. Monitoring of these components or resources requires the deployment of a monitoring system across the entire Kubernetes cluster dimension. - -### Monitor the host - -Monitoring the host and its resources works in the same way as monitoring physical resources of a traditional server. - -If you already have a monitoring system for your physical server in your existing infrastructure, you only need to add the host that holds Kubernetes to the existing monitoring system by conventional means; if there is no monitoring system available, or you want to deploy a separate monitoring system to monitor the host that holds Kubernetes, then you can use any monitoring system that you are familiar with. - -The newly deployed monitoring system can run on a separate server, directly on the host that holds Kubernetes, or in a Kubernetes cluster. Different deployment methods might mean differences in the deployment configuration and resource utilization, but there are no major differences in usage. - -Some common open source monitoring systems that can be used to monitor server resources are: - -- [CollectD](https://collectd.org/) -- [Nagios](https://www.nagios.org/) -- [Prometheus](http://prometheus.io/) & [node_exporter](https://github.com/prometheus/node_exporter) -- [Zabbix](https://www.zabbix.com/) - -Some cloud service providers or specialized performance monitoring service providers also have their own free or chargeable monitoring solutions that you can choose from. - -It is recommended to deploy a host monitoring system in the Kubernetes cluster via [Prometheus Operator](https://github.com/coreos/prometheus-operator) based on [Node Exporter](https://github.com/prometheus/node_exporter) and Prometheus. This solution can also be compatible with and used for monitoring the Kubernetes' own components. +You can also set `spec.prometheus.service.type` to `NodePort` or `LoadBalancer`, and then view the monitoring data through `NodePort` or `LoadBalancer`. + +### Set kube-prometheus and AlertManager + +In some cases, TidbMonitor needs to obtain the monitoring metrics on Kubernetes. To obtain the [kube-prometheus](https://github.com/coreos/kube-prometheus) metrics, configure `TidbMonitor.Spec.kubePrometheusURL`. + +Similarly, you can configure TidbMonitor to push the monitoring alert to [AlertManager](https://prometheus.io/docs/alerting/alertmanager/). + +```yaml +apiVersion: pingcap.com/v1alpha1 +kind: TidbMonitor +metadata: + name: basic +spec: + clusters: + - name: basic + kubePrometheusURL: "your-kube-prometheus-url" + alertmanagerURL: "your-alert-manager-url" + prometheus: + baseImage: prom/prometheus + version: v2.18.1 + service: + type: NodePort + grafana: + baseImage: grafana/grafana + version: 6.1.6 + service: + type: NodePort + initializer: + baseImage: pingcap/tidb-monitor-initializer + version: v4.0.9 + reloader: + baseImage: pingcap/tidb-monitor-reloader + version: v1.0.1 + imagePullPolicy: IfNotPresent +``` -### Monitor Kubernetes components +## Enable Ingress + +This section introduces how to enable Ingress for TidbMonitor. [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) is an API object that exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. + +### Prerequisites + +Before using Ingress, you need to install the Ingress controller. Simply creating the Ingress resource does not take effect. + +You need to deploy the [NGINX Ingress controller](https://kubernetes.github.io/ingress-nginx/deploy/), or choose from various [Ingress controllers](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/). + +For more information, see [Ingress Prerequisites](https://kubernetes.io/docs/concepts/services-networking/ingress/#prerequisites). + +### Access TidbMonitor using Ingress + +Currently, TidbMonitor provides a method to expose the Prometheus/Grafana service through Ingress. For details about Ingress, see [Ingress official documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/). + +The following example shows how to enable Prometheus and Grafana in TidbMonitor: + +```yaml +apiVersion: pingcap.com/v1alpha1 +kind: TidbMonitor +metadata: + name: ingress-demo +spec: + clusters: + - name: demo + persistent: false + prometheus: + baseImage: prom/prometheus + version: v2.18.1 + ingress: + hosts: + - example.com + annotations: + foo: "bar" + grafana: + baseImage: grafana/grafana + version: 6.1.6 + service: + type: ClusterIP + ingress: + hosts: + - example.com + annotations: + foo: "bar" + initializer: + baseImage: pingcap/tidb-monitor-initializer + version: v4.0.9 + reloader: + baseImage: pingcap/tidb-monitor-reloader + version: v1.0.1 + imagePullPolicy: IfNotPresent +``` -For monitoring Kubernetes components, you can refer to the solutions provided in the [Kubernetes official documentation](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/) or use other Kubernetes-compatible monitoring systems. +To modify the setting of Ingress Annotations, configure `spec.prometheus.ingress.annotations` and `spec.grafana.ingress.annotations`. If you use the default Nginx Ingress, see [Nginx Ingress Controller Annotation](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/) for details. + +The TidbMonitor Ingress setting also supports TLS. The following example shows how to configure TLS for Ingress. See [Ingress TLS](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls) for details. + +```yaml +apiVersion: pingcap.com/v1alpha1 +kind: TidbMonitor +metadata: + name: ingress-demo +spec: + clusters: + - name: demo + persistent: false + prometheus: + baseImage: prom/prometheus + version: v2.18.1 + ingress: + hosts: + - example.com + tls: + - hosts: + - example.com + secretName: testsecret-tls + grafana: + baseImage: grafana/grafana + version: 6.1.6 + service: + type: ClusterIP + initializer: + baseImage: pingcap/tidb-monitor-initializer + version: v4.0.9 + reloader: + baseImage: pingcap/tidb-monitor-reloader + version: v1.0.1 + imagePullPolicy: IfNotPresent +``` -Some cloud service providers may provide their own solutions for monitoring Kubernetes components, and some specialized performance monitoring service providers have their own Kubernetes integration solutions that you can choose from. +TLS Secret must include the `tls.crt` and `tls.key` keys, which include the certificate and private key used for TLS. For example: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: testsecret-tls + namespace: ${namespace} +data: + tls.crt: base64 encoded cert + tls.key: base64 encoded key +type: kubernetes.io/tls +``` -TiDB Operator is actually a container running in Kubernetes. For this reason, you can monitor TiDB Operator by choosing any monitoring system that can monitor the status and resources of a Kubernetes container without deploying additional monitoring components. +In a public cloud-deployed Kubernetes cluster, you can usually [configure Loadbalancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/) to access Ingress through a domain name. If you cannot configure the Loadbalancer service (for example, when you use NodePort as the service type of Ingress), you can access the service in a way equivalent to the following command: -It is recommended to deploy a host monitoring system via [Prometheus Operator](https://github.com/coreos/prometheus-operator) based on [Node Exporter](https://github.com/prometheus/node_exporter) and Prometheus. This solution can also be compatible with and used for monitoring host resources. +```shell +curl -H "Host: example.com" ${node_ip}:${NodePort} +``` ## Configure alert -### Alerts in the TiDB Cluster - When Prometheus is deployed with a TiDB cluster, some default alert rules are automatically imported. You can view all alert rules and statuses in the current system by accessing the Alerts page of Prometheus through a browser. The custom configuration of alert rules is supported. You can modify the alert rules by taking the following steps: -1. When deploying the monitoring system for the TiDB cluster, set `spec.reloader.service.type` to `NodePort` or `LoadBalancer`. For details, see [TidbMonitor Configuration](monitor-using-tidbmonitor.md). +1. When deploying the monitoring system for the TiDB cluster, set `spec.reloader.service.type` to `NodePort` or `LoadBalancer`. 2. Access the `reloader` service through `NodePort` or `LoadBalancer`. Click the `Files` button above to select the alert rule file to be modified, and make the custom configuration. Click `Save` after the modification. The default Prometheus and alert configuration do not support sending alert messages. To send an alert message, you can integrate Prometheus with any tool that supports Prometheus alerts. It is recommended to manage and send alert messages via [AlertManager](https://prometheus.io/docs/alerting/alertmanager/). -- If you already have an available AlertManager service in your existing infrastructure, you can set the value of `spec.alertmanagerURL` to the address of `AlertManager`, which will be used by Prometheus. For details, refer to [Set kube-prometheus and AlertManager](monitor-using-tidbmonitor.md#set-kube-prometheus-and-alertmanager). +- If you already have an available AlertManager service in your existing infrastructure, you can set the value of `spec.alertmanagerURL` to the address of `AlertManager`, which will be used by Prometheus. For details, refer to [Set kube-prometheus and AlertManager](#set-kube-prometheus-and-alertmanager). - If no AlertManager service is available, or if you want to deploy a separate AlertManager service, refer to the [Prometheus official document](https://github.com/prometheus/alertmanager). - -### Alerts in Kubernetes - -If you deploy a monitoring system for Kubernetes hosts and services by Prometheus Operator, some alert rules are configured by default, and an AlertManager service is deployed. For details, see [kube-prometheus](https://github.com/coreos/). - -If you monitor Kubernetes hosts and services by using other tools or services, you can consult the corresponding information provided by the tool or service provider. diff --git a/en/monitor-kubernetes.md b/en/monitor-kubernetes.md new file mode 100644 index 0000000000..aff309ba49 --- /dev/null +++ b/en/monitor-kubernetes.md @@ -0,0 +1,47 @@ +--- +title: Monitoring and Alerts on Kubernetes +summary: Learn the monitoring and alerts on Kubernetes. +--- + +# Monitoring and Alerts on Kubernetes + +This document describes how to monitor a Kubernetes cluster and configure alerts for the cluster. + +## Monitor the Kubernetes cluster + +The TiDB monitoring system deployed with the cluster only focuses on the operation of the TiDB components themselves, and does not include the monitoring of container resources, hosts, Kubernetes components, or TiDB Operator. To monitor these components or resources, you need to deploy a monitoring system across the entire Kubernetes cluster. + +### Monitor the host + +Monitoring the host and its resources works in the same way as monitoring physical resources of a traditional server. + +If you already have a monitoring system for your physical server in your existing infrastructure, you only need to add the host that holds Kubernetes to the existing monitoring system by conventional means; if there is no monitoring system available, or if you want to deploy a separate monitoring system to monitor the host that holds Kubernetes, then you can use any monitoring system that you are familiar with. + +The newly deployed monitoring system can run on a separate server, directly on the host that holds Kubernetes, or in a Kubernetes cluster. Different deployment methods might have differences in the deployment configuration and resource utilization, but there are no major differences in usage. + +Some common open source monitoring systems that can be used to monitor server resources are: + +- [CollectD](https://collectd.org/) +- [Nagios](https://www.nagios.org/) +- [Prometheus](http://prometheus.io/) & [node_exporter](https://github.com/prometheus/node_exporter) +- [Zabbix](https://www.zabbix.com/) + +Some cloud service providers or specialized performance monitoring service providers also have their own free or chargeable monitoring solutions that you can choose from. + +It is recommended to deploy a host monitoring system in the Kubernetes cluster via [Prometheus Operator](https://github.com/coreos/prometheus-operator) based on [Node Exporter](https://github.com/prometheus/node_exporter) and Prometheus. This solution can also be compatible with and used for monitoring the Kubernetes' own components. + +### Monitor Kubernetes components + +For monitoring Kubernetes components, you can refer to the solutions provided in the [Kubernetes official documentation](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/) or use other Kubernetes-compatible monitoring systems. + +Some cloud service providers may provide their own solutions for monitoring Kubernetes components. Some specialized performance monitoring service providers have their own Kubernetes integration solutions that you can choose from. + +TiDB Operator is actually a container running in Kubernetes. For this reason, you can monitor TiDB Operator by choosing any monitoring system that can monitor the status and resources of a Kubernetes container without deploying additional monitoring components. + +It is recommended to deploy a host monitoring system via [Prometheus Operator](https://github.com/coreos/prometheus-operator) based on [Node Exporter](https://github.com/prometheus/node_exporter) and Prometheus. This solution can also be compatible with and used for monitoring host resources. + +### Alerts in Kubernetes + +If you deploy a monitoring system for Kubernetes hosts and services using Prometheus Operator, some alert rules are configured by default, and an AlertManager service is deployed. For details, see [kube-prometheus](https://github.com/coreos/kube-prometheus). + +If you monitor Kubernetes hosts and services by using other tools or services, you can consult the corresponding information provided by the tool or service provider. diff --git a/en/monitor-using-tidbmonitor.md b/en/monitor-using-tidbmonitor.md deleted file mode 100644 index f2b91de9c7..0000000000 --- a/en/monitor-using-tidbmonitor.md +++ /dev/null @@ -1,301 +0,0 @@ ---- -title: Monitor a TiDB Cluster Using TidbMonitor -summary: This document introduces how to monitor a TiDB cluster using TidbMonitor. -aliases: ['/docs/tidb-in-kubernetes/dev/monitor-using-tidbmonitor/'] ---- - -# Monitor a TiDB Cluster Using TidbMonitor - -In TiDB Operator v1.1 or later versions, you can monitor a TiDB cluster on a Kubernetes cluster by using a simple Custom Resource (CR) file called TidbMonitor. - -## Quick start - -### Prerequisites - -- TiDB Operator **v1.1.0** (or later versions) is installed, and the related Custom Resource Definition (CRD) file is updated. - -- The default storageClass is set, which has enough Persistent Volumes (PV). 6 PVs are needed by default. You can check this setting by executing the following command: - - {{< copyable "shell-regular" >}} - - ```shell - kubectl get storageClass - ``` - - ```shell - NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE - standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 14h - ``` - -### Install - -To deploy a TidbMonitor component, save the following content as a `.yaml` file and execute the `kubectl apply -f` command. - -```yaml -apiVersion: pingcap.com/v1alpha1 -kind: TidbMonitor -metadata: - name: basic -spec: - clusters: - - name: basic - prometheus: - baseImage: prom/prometheus - version: v2.18.1 - service: - type: NodePort - grafana: - baseImage: grafana/grafana - version: 6.1.6 - initializer: - baseImage: pingcap/tidb-monitor-initializer - version: v4.0.9 - reloader: - baseImage: pingcap/tidb-monitor-reloader - version: v1.0.1 - imagePullPolicy: IfNotPresent -``` - -> **Note:** -> -> * One TidbMonitor can monitor only one TidbCluster. -> * `spec.clusters[0].name` needs to be set to the TidbCluster name of your TiDB Cluster. - -You can also quickly deploy TidbMonitor using the following command: - -{{< copyable "shell-regular" >}} - -```shell -kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/examples/basic/tidb-monitor.yaml -n ${namespace} -``` - -If the server does not have an external network, refer to [Deploy the TiDB cluster](deploy-on-general-kubernetes.md#deploy-the-tidb-cluster) to download the required Docker image on the machine with an external network and upload it to the server. - -After the deployment is finished, you can check whether TidbMonitor is started by executing the `kubectl get pod` command: - -{{< copyable "shell-regular" >}} - -```shell -kubectl get pod -l app.kubernetes.io/instance=basic -n ${namespace} | grep monitor -``` - -``` -basic-monitor-85fcf66bc4-cwpcn 3/3 Running 0 117s -``` - -### View the monitoring dashboard - -To view the monitoring dashboard, execute the following command: - -{{< copyable "shell-regular" >}} - -```shell -kubectl -n ${namespace} port-forward svc/basic-grafana 3000:3000 &>/tmp/pf-grafana.log & -``` - -Visit `localhost:3000` to access the dashboard. - -### Delete the monitor - -{{< copyable "shell-regular" >}} - -```shell -kubectl delete tidbmonitor basic -n ${namespace} -``` - -## Persist the monitoring data - -If you want to persist the monitoring data in TidbMonitor, enable the `persist` option in TidbMonitor: - -```yaml -apiVersion: pingcap.com/v1alpha1 -kind: TidbMonitor -metadata: - name: basic -spec: - clusters: - - name: basic - persistent: true - storageClassName: ${storageClassName} - storage: 5G - prometheus: - baseImage: prom/prometheus - version: v2.18.1 - service: - type: NodePort - grafana: - baseImage: grafana/grafana - version: 6.1.6 - service: - type: NodePort - initializer: - baseImage: pingcap/tidb-monitor-initializer - version: v4.0.9 - reloader: - baseImage: pingcap/tidb-monitor-reloader - version: v1.0.1 - imagePullPolicy: IfNotPresent -``` - -To verify the status of Persistent Volume Claim (PVC), execute the following command: - -{{< copyable "shell-regular" >}} - -```shell -kubectl get pvc -l app.kubernetes.io/instance=basic,app.kubernetes.io/component=monitor -n ${namespace} -``` - -``` -NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE -basic-monitor Bound pvc-6db79253-cc9e-4730-bbba-ba987c29db6f 5G RWO standard 51s -``` - -### Set kube-prometheus and AlertManager - -In some cases, TidbMonitor needs to obtain the monitoring metrics on Kubernetes. To obtain the kube-prometheus metrics, configure `TidbMonitor.Spec.kubePrometheusURL`. For details, see [kube-prometheus](https://github.com/coreos/kube-prometheus). - -Similarly, you can configure TidbMonitor to push the monitoring alert to AlertManager. For details, see [AlertManager](https://prometheus.io/docs/alerting/alertmanager/). - -```yaml -apiVersion: pingcap.com/v1alpha1 -kind: TidbMonitor -metadata: - name: basic -spec: - clusters: - - name: basic - kubePrometheusURL: "your-kube-prometheus-url" - alertmanagerURL: "your-alert-manager-url" - prometheus: - baseImage: prom/prometheus - version: v2.18.1 - service: - type: NodePort - grafana: - baseImage: grafana/grafana - version: 6.1.6 - service: - type: NodePort - initializer: - baseImage: pingcap/tidb-monitor-initializer - version: v4.0.9 - reloader: - baseImage: pingcap/tidb-monitor-reloader - version: v1.0.1 - imagePullPolicy: IfNotPresent -``` - -## Enable Ingress - -This section introduces how to enable Ingress for TidbMonitor. Ingress is an API object that exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. - -### Prerequisites - -Before using Ingress, you need to install the Ingress controller. Simply creating the Ingress resource does not take effect. - -You need to deploy the [NGINX Ingress controller](https://kubernetes.github.io/ingress-nginx/deploy/), or choose from various [Ingress controllers](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/). - -For more information, see [Ingress Prerequisites](https://kubernetes.io/docs/concepts/services-networking/ingress/#prerequisites). - -### Access TidbMonitor using Ingress - -Currently, TidbMonitor provides a method to expose the Prometheus/Grafana service through Ingress. For details about Ingress, see [Ingress official documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/). - -The following example shows how to enable Prometheus and Grafana in TidbMonitor: - -```yaml -apiVersion: pingcap.com/v1alpha1 -kind: TidbMonitor -metadata: - name: ingress-demo -spec: - clusters: - - name: demo - persistent: false - prometheus: - baseImage: prom/prometheus - version: v2.18.1 - ingress: - hosts: - - example.com - annotations: - foo: "bar" - grafana: - baseImage: grafana/grafana - version: 6.1.6 - service: - type: ClusterIP - ingress: - hosts: - - example.com - annotations: - foo: "bar" - initializer: - baseImage: pingcap/tidb-monitor-initializer - version: v4.0.9 - reloader: - baseImage: pingcap/tidb-monitor-reloader - version: v1.0.1 - imagePullPolicy: IfNotPresent -``` - -To modify the setting of Ingress Annotations, configure `spec.prometheus.ingress.annotations` and `spec.grafana.ingress.annotations`. If you use the default Nginx Ingress, see [Nginx Ingress Controller Annotation](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/) for details. - -The TidbMonitor Ingress setting also supports TLS. The following example shows how to configure TLS for Ingress. See [Ingress TLS](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls) for details. - -```yaml -apiVersion: pingcap.com/v1alpha1 -kind: TidbMonitor -metadata: - name: ingress-demo -spec: - clusters: - - name: demo - persistent: false - prometheus: - baseImage: prom/prometheus - version: v2.18.1 - ingress: - hosts: - - example.com - tls: - - hosts: - - example.com - secretName: testsecret-tls - grafana: - baseImage: grafana/grafana - version: 6.1.6 - service: - type: ClusterIP - initializer: - baseImage: pingcap/tidb-monitor-initializer - version: v4.0.9 - reloader: - baseImage: pingcap/tidb-monitor-reloader - version: v1.0.1 - imagePullPolicy: IfNotPresent -``` - -TLS Secret must include the `tls.crt` and `tls.key` keys, which include the certificate and private key used for TLS. For example: - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: testsecret-tls - namespace: ${namespace} -data: - tls.crt: base64 encoded cert - tls.key: base64 encoded key -type: kubernetes.io/tls -``` - -In a public cloud-deployed Kubernetes cluster, you can usually [configure Loadbalancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/) to access Ingress through a domain name. If you cannot configure the Loadbalancer service (for example, when you use NodePort as the service type of Ingress), you can access the service in a way equivalent to the following command: - -```shell -curl -H "Host: example.com" ${node_ip}:${NodePort} -``` - -## References - -For more detailed API information of TidbMonitor, see [TiDB Operator API documentation](https://github.com/pingcap/tidb-operator/blob/master/docs/api-references/docs.md). diff --git a/en/notes-tidb-operator-v1.1.md b/en/notes-tidb-operator-v1.1.md index 78bb532650..2883ebd278 100644 --- a/en/notes-tidb-operator-v1.1.md +++ b/en/notes-tidb-operator-v1.1.md @@ -48,7 +48,7 @@ Since TiDB Operator v1.1, you can configure TiDB, TiKV, and PD directly in TidbC ### Monitor -To create the TidbMonitor CR and manage the Monitor component, refer to [Monitor a TiDB Cluster using TidbMonitor](monitor-using-tidbmonitor.md). +To create the TidbMonitor CR and manage the Monitor component, refer to [Monitor a TiDB Cluster](monitor-a-tidb-cluster.md). > **Note:** > diff --git a/zh/monitor-a-tidb-cluster.md b/zh/monitor-a-tidb-cluster.md index d6e39126aa..60826bd13b 100644 --- a/zh/monitor-a-tidb-cluster.md +++ b/zh/monitor-a-tidb-cluster.md @@ -68,7 +68,9 @@ NAME STATUS VOLUME CAPACITY A basic-monitor Bound pvc-6db79253-cc9e-4730-bbba-ba987c29db6f 5G RWO standard 51s ``` -### 自定义Prometheus 配置 +### 自定义 Prometheus 配置 + +用户可以通过自定义配置文件或增加额外的命令行参数，来自定义 Prometheus 配置。 #### 使用自定义配置文件 @@ -163,6 +165,8 @@ spec: ## 开启 Ingress +本节介绍如何为 TidbMonitor 开启 Ingress。[Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) 是一个 API 对象，负责管理集群中服务的外部访问。 + ### 环境准备 使用 `Ingress` 前，需要在 Kubernetes 集群中安装 `Ingress` 控制器，否则仅创建 `Ingress` 资源无效。你可能需要部署 `Ingress` 控制器，例如 [ingress-nginx](https://kubernetes.github.io/ingress-nginx/deploy/)。你可以从许多 [Ingress 控制器](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/)中进行选择。 @@ -270,8 +274,6 @@ curl -H "Host: example.com" ${node_ip}:${NodePort} ## 告警配置 -### TiDB 集群告警 - 在随 TiDB 集群部署 Prometheus 时，会自动导入一些默认的告警规则，可以通过浏览器访问 Prometheus 的 Alerts 页面查看当前系统中的所有告警规则和状态。 目前支持自定义配置告警规则，可以参考下面步骤修改告警规则：