tiflash: add the maintain.md doc #2189

toutdesuite · 2020-04-07T12:08:43Z

What is changed, added or deleted? (Required)

add the maintain.md doc

Which TiDB version(s) do your changes apply to? (Required)

If you select two or more versions from above, to trigger the bot to cherry-pick this PR to your desired release version branch(es), you must add corresponding labels such as needs-cherry-pick-4.0, needs-cherry-pick-3.1, needs-cherry-pick-3.0, and needs-cherry-pick-2.1.

What is the related PR or file link(s)?

This PR is translated from:reference: add 11 new documents for TiFlash docs-cn#2603
Other reference link(s):

toutdesuite · 2020-04-07T12:39:48Z

reference/tiflash/maintain.md

+
+## Logout a TiFlash node
+
+Logouting a TiFlash node differs from [Scaling in the TiFlash node](/reference/tiflash/scale.md#scale-in-tiflash-node) in that the logout doesn't remove the node from TiDB Ansible; instead, it just safely shutdown the process.


The reference anchor is not finalized because scale.md is not merged

Please take a look at L90. The anchor is decided: #scale-in-a-tiflash-node.

ilovesoup

LGTM

yikeke · 2020-04-08T10:02:06Z

reference/tiflash/maintain.md

+
+# Maintain a TiFlash Cluster
+
+This document describes common operations when you maintain a TiFlash cluster, including checking the version, node logout, troubleshooting, critical logs, and a system table.


Suggested change

This document describes common operations when you maintain a TiFlash cluster, including checking the version, node logout, troubleshooting, critical logs, and a system table.

This document describes how to perform common operations when you maintain a TiFlash cluster, including checking the TiFlash version, taking TiFlash nodes down, and troubleshooting TiFlash. This document also introduces critical logs and system tables of TiFlash.

yikeke · 2020-04-08T10:06:57Z

reference/tiflash/maintain.md

+    LD_LIBRARY_PATH=./ ./tiflash version
+    ```
+
+- Check the TiFlash version by referring to the TiFlash log. For the log path, see the [logger] part in [Configure the `tiflash.toml` file](/reference/tiflash/configuration.md#configure-the-`tiflash.toml`-file). For example:


Suggested change

- Check the TiFlash version by referring to the TiFlash log. For the log path, see the [logger] part in [Configure the `tiflash.toml` file](/reference/tiflash/configuration.md#configure-the-`tiflash.toml`-file). For example:

- Check the TiFlash version by referring to the TiFlash log. For the log path, see the `[logger]` part in [the `tiflash.toml` configuration file](/reference/tiflash/configuration.md#configure-the-`tiflash.toml`-file). For example:

The anchor link is wrong. Please correct it by referring to the PingCAP documentation style guide. @toutdesuite

yikeke · 2020-04-08T10:15:13Z

reference/tiflash/maintain.md

+    <information>: TiFlash version: TiFlash 0.2.0 master-375035282451103999f3863c691e2fc2
+    ```
+
+## Logout a TiFlash node


Suggested change

## Logout a TiFlash node

## Take a TiFlash node down

take nodes down is an expression used more commonly in our technical documents and it has more search results in Google. Please update this expression in other places (if any). @toutdesuite

reference/tiflash/maintain.md

…into maintain-tiflash

yikeke · 2020-04-08T11:24:23Z

reference/tiflash/maintain.md

+
+## Take a TiFlash node down
+
+Taking a TiFlash node down differs from [Scaling in a TiFlash node](/reference/tiflash/scale.md#scale-in-a-tiflash-node) in that the former doesn't remove the node from TiDB Ansible; instead, it just safely shutdown the process.


Suggested change

Taking a TiFlash node down differs from [Scaling in a TiFlash node](/reference/tiflash/scale.md#scale-in-a-tiflash-node) in that the former doesn't remove the node from TiDB Ansible; instead, it just safely shutdown the process.

Taking a TiFlash node down differs from [Scaling in a TiFlash node](/reference/tiflash/scale.md#scale-in-a-tiflash-node) in that the former doesn't remove the node in TiDB Ansible; instead, it just safely shuts down the TiFlash process.

yikeke · 2020-04-08T11:29:04Z

reference/tiflash/maintain.md

+>
+> After you take the TiFlash node down, if the number of the remaining nodes in the TiFlash cluster is greater than or equal to the maximum replicas of all data tables, you can go directly to step 3.
+
+1. For a TiDB server, if the number of replicas of tables is greater than or equal to that of the remaining TiFlash nodes in the cluster, execute the following command:


Suggested change

1. For a TiDB server, if the number of replicas of tables is greater than or equal to that of the remaining TiFlash nodes in the cluster, execute the following command:

1. If the number of replicas of tables is greater than or equal to that of the remaining TiFlash nodes in the cluster, execute the following command on those tables in the TiDB client:

yikeke · 2020-04-08T11:30:46Z

reference/tiflash/maintain.md

+    alter table <db-name>.<table-name> set tiflash replica 0;
+    ```
+
+2. To ensure TiFlash replicas of related tables are removed, see [View the Table Replication Progress](/reference/tiflash/use-tiflash.md#view-the-table-replication-progress). If you cannot view the replication progress of the related tables, it means that the replicas are removed.


Suggested change

2. To ensure TiFlash replicas of related tables are removed, see [View the Table Replication Progress](/reference/tiflash/use-tiflash.md#view-the-table-replication-progress). If you cannot view the replication progress of the related tables, it means that the replicas are removed.

2. To ensure that the TiFlash replicas of these tables are removed, see [View the Table Replication Progress](/reference/tiflash/use-tiflash.md#view-the-table-replication-progress). If you cannot view the replication progress of the tables, it means that the replicas are removed.

yikeke · 2020-04-08T11:32:04Z

reference/tiflash/maintain.md

+
+2. To ensure TiFlash replicas of related tables are removed, see [View the Table Replication Progress](/reference/tiflash/use-tiflash.md#view-the-table-replication-progress). If you cannot view the replication progress of the related tables, it means that the replicas are removed.
+
+3. Input the `store` command into [pd-ctl](/reference/tools/pd-control.md) (the binary file in `resources/bin` in the tidb-ansible directory) to view the `store id` of the TiFlash node.


Suggested change

3. Input the `store` command into [pd-ctl](/reference/tools/pd-control.md) (the binary file in `resources/bin` in the tidb-ansible directory) to view the `store id` of the TiFlash node.

3. Input the `store` command into [pd-ctl](/reference/tools/pd-control.md) (the binary file is in `resources/bin` of the tidb-ansible directory) to view the `store id` of the TiFlash node.

yikeke · 2020-04-08T11:34:35Z

reference/tiflash/maintain.md

+
+4. Input `store delete <store_id>` into `pd-ctl`. Here `<store_id>` refers to the `store id` in step 3.
+
+5. When the corresponding `store` of the node disappeared, or when `state_name` is changed to `Tomestone`, shutdown the TiFlash process.


Suggested change

5. When the corresponding `store` of the node disappeared, or when `state_name` is changed to `Tomestone`, shutdown the TiFlash process.

5. When the corresponding `store` of the node disappears, or when `state_name` is changed to `Tomestone`, stop the TiFlash process.

In very few cases we use the past tense in our technical documents. Please read the Google Developer Style Guide for reference.

yikeke · 2020-04-08T11:36:32Z

reference/tiflash/maintain.md

+
+> **Note:**
+>
+> If you don't cancel all tables replicated to TiFlash before all TiFlash nodes in a cluster stop running, you need to manually delete the replication rule in PD. Or you cannot successfully take the TiFlash node down.


Suggested change

> If you don't cancel all tables replicated to TiFlash before all TiFlash nodes in a cluster stop running, you need to manually delete the replication rule in PD. Or you cannot successfully take the TiFlash node down.

> If you don't cancel all tables replicated to TiFlash before all TiFlash nodes stop running, you need to manually delete the replication rule in PD. Or you cannot successfully take the TiFlash node down.

yikeke · 2020-04-08T11:37:37Z

reference/tiflash/maintain.md

+
+## TiFlash troubleshooting
+
+This section describes some common questions of TiFlash, the reasons, and the solutions.


Suggested change

This section describes some common questions of TiFlash, the reasons, and the solutions.

This section describes some commonly encountered issues when using TiFlash, the reasons, and the solutions.

yikeke · 2020-04-08T11:38:21Z

reference/tiflash/maintain.md

+
+This section describes some common questions of TiFlash, the reasons, and the solutions.
+
+### TiFlash replica is always in an unusable state


Suggested change

### TiFlash replica is always in an unusable state

### TiFlash replica is always unavailable

yikeke · 2020-04-08T11:41:35Z

reference/tiflash/maintain.md

+
+### TiFlash replica is always in an unusable state
+
+This is because TiFlash is in the exception status caused by the configuration error or the environment problems. You can take the following steps to identify the problem component:


Suggested change

This is because TiFlash is in the exception status caused by the configuration error or the environment problems. You can take the following steps to identify the problem component:

This is because TiFlash is in an abnormal state caused by configuration errors or environment issues. Take the following steps to identify the faulty component:

yikeke · 2020-04-08T11:45:23Z

reference/tiflash/maintain.md

+
+    The expected result is `"enable-placement-rules": "true"`.
+
+2. Check whether the TiFlash process in the operation system is working correctly using `UpTime` of the TiFlash-Summary monitor panel.


Suggested change

2. Check whether the TiFlash process in the operation system is working correctly using `UpTime` of the TiFlash-Summary monitor panel.

2. Check whether the TiFlash process is working correctly by viewing `UpTime` on the TiFlash-Summary monitoring panel.

yikeke · 2020-04-08T11:48:53Z

reference/tiflash/maintain.md

+    echo "store" | /path/to/pd-ctl -u http://<pd-ip>:<pd-port>
+    ```
+
+    If `store.labels` includes information such as `{"key": "engine", "value": "tiflash"}`, it refers to the TiFlash proxy.


Suggested change

If `store.labels` includes information such as `{"key": "engine", "value": "tiflash"}`, it refers to the TiFlash proxy.

The TiFlash proxy's `store.labels` includes information such as `{"key": "engine", "value": "tiflash"}`. You can check this information to confirm a TiFlash proxy.

Please review my suggestion. The Chinese is: store.labels 中含有 {"key": "engine", "value": "tiflash"} 信息的为 TiFlash proxy。 @ilovesoup

yikeke · 2020-04-08T11:59:31Z

reference/tiflash/maintain.md

+
+    If `store.labels` includes information such as `{"key": "engine", "value": "tiflash"}`, it refers to the TiFlash proxy.
+
+4. Check whether `pd buddy` can print the logs correctly (the value of `log` in the [flash.flash_cluster] configuration item of the log path, is by default the `tmp` directory configured by the TiFlash configuration file).


Suggested change

4. Check whether `pd buddy` can print the logs correctly (the value of `log` in the [flash.flash_cluster] configuration item of the log path, is by default the `tmp` directory configured by the TiFlash configuration file).

4. Check whether `pd buddy` can correctly print the logs (the log path is the value of `log` in the [flash.flash_cluster] configuration item; the default log path is under the `tmp` directory configured in the TiFlash configuration file).

原文是“查看 pd buddy 是否正常打印日志（日志路径的对应配置项 [flash.flash_cluster] log 设置的值，默认为 TiFlash 配置文件配置的 tmp 目录下）”，完整的说法是不是“查看 pd buddy 是否正常打印日志（日志路径为对应配置项 [flash.flash_cluster] 中 log 设置的值，日志路径默认在 TiFlash 配置文件配置的 tmp 目录下）”？ @ilovesoup

yikeke · 2020-04-08T12:01:30Z

reference/tiflash/maintain.md

+
+    Reconfirm the value of `max-replicas`.
+
+6. Check whether the remaining disk space of the machine (where `store` of the TiFlash node is) is sufficient. By default, when the remaining disk space is less than 20% of the `store` capacity (which is controlled by the `low-space-ratio` parameter), PD cannot schedule data to TiFlash.


Suggested change

6. Check whether the remaining disk space of the machine (where `store` of the TiFlash node is) is sufficient. By default, when the remaining disk space is less than 20% of the `store` capacity (which is controlled by the `low-space-ratio` parameter), PD cannot schedule data to TiFlash.

6. Check whether the remaining disk space of the machine (where `store` of the TiFlash node is) is sufficient. By default, when the remaining disk space is less than 20% of the `store` capacity (which is controlled by the `low-space-ratio` parameter), PD cannot schedule data to this TiFlash node.

yikeke · 2020-04-08T12:03:40Z

reference/tiflash/maintain.md

+
+### TiFlash query time is unstable, and error log prints many `Lock Exception` messages
+
+This is because large amounts of data are written to the cluster, which leads to the situation that the TiFlash query encounters a lock and requires query retry.


Suggested change

This is because large amounts of data are written to the cluster, which leads to the situation that the TiFlash query encounters a lock and requires query retry.

This is because large amounts of data are written to the cluster, which causes that the TiFlash query encounters a lock and requires query retry.

yikeke · 2020-04-08T12:08:28Z

reference/tiflash/maintain.md

+
+This is because large amounts of data are written to the cluster, which leads to the situation that the TiFlash query encounters a lock and requires query retry.
+
+You can set the query timestamp to one second earlier in TiDB (for example, `set @@tidb_snapshot=412881237115666555;`), to reduce the possibility that TiFlash query encounters a lock; thereby mitigating the risk of unstable query time.


Suggested change

You can set the query timestamp to one second earlier in TiDB (for example, `set @@tidb_snapshot=412881237115666555;`), to reduce the possibility that TiFlash query encounters a lock; thereby mitigating the risk of unstable query time.

You can set the query timestamp to one second earlier in TiDB (for example, `set @@tidb_snapshot=412881237115666555;`). This makes less TiFlash queries encounter a lock and mitigate the risk of unstable query time.

yikeke · 2020-04-08T12:09:11Z

reference/tiflash/maintain.md

+
+You can set the query timestamp to one second earlier in TiDB (for example, `set @@tidb_snapshot=412881237115666555;`), to reduce the possibility that TiFlash query encounters a lock; thereby mitigating the risk of unstable query time.
+
+### Partial queries return `Region Unavailable`


Suggested change

### Partial queries return `Region Unavailable`

### Some queries return the `Region Unavailable` error

yikeke · 2020-04-08T12:10:32Z

reference/tiflash/maintain.md

+
+### Partial queries return `Region Unavailable`
+
+If the load pressure in TiFlash is so heavy that TiFlash data replication falls behind. Some queries might return error message `Region Unavailable`.


Suggested change

If the load pressure in TiFlash is so heavy that TiFlash data replication falls behind. Some queries might return error message `Region Unavailable`.

If the load pressure on TiFlash is too heavy and it causes that TiFlash data replication falls behind, some queries might return the `Region Unavailable` error.

yikeke · 2020-04-08T12:12:07Z

reference/tiflash/maintain.md

+
+If the load pressure in TiFlash is so heavy that TiFlash data replication falls behind. Some queries might return error message `Region Unavailable`.
+
+In this case, you can share the pressure by adding TiFlash nodes.


Suggested change

In this case, you can share the pressure by adding TiFlash nodes.

In this case, you can balance the load pressure by adding more TiFlash nodes.

yikeke · 2020-04-08T12:20:57Z

reference/tiflash/maintain.md

+| Log Information | Log Description |
+|---------------|-------------------|
+| [ 23 ] <Information> KVStore: Start to persist [region 47, applied: term 6 index 10] | Data starts to be replicated (the number in the square brackets at the start of the log refers to the thread ID |
+| [ 30 ] <Debug> CoprocessorHandler: grpc::Status DB::CoprocessorHandler::execute() | `Handling DAG request` refers to that TiFlash starts to handle a Coprocessor request |


Suggested change

| [ 30 ] <Debug> CoprocessorHandler: grpc::Status DB::CoprocessorHandler::execute() | `Handling DAG request` refers to that TiFlash starts to handle a Coprocessor request |

| [ 30 ] <Debug> CoprocessorHandler: grpc::Status DB::CoprocessorHandler::execute() | Handling DAG request, that is, TiFlash starts to handle a Coprocessor request |

yikeke · 2020-04-08T12:21:37Z

reference/tiflash/maintain.md

+|---------------|-------------------|
+| [ 23 ] <Information> KVStore: Start to persist [region 47, applied: term 6 index 10] | Data starts to be replicated (the number in the square brackets at the start of the log refers to the thread ID |
+| [ 30 ] <Debug> CoprocessorHandler: grpc::Status DB::CoprocessorHandler::execute() | `Handling DAG request` refers to that TiFlash starts to handle a Coprocessor request |
+| [ 30 ] <Debug> CoprocessorHandler: grpc::Status DB::CoprocessorHandler::execute() | `Handle DAG request done` refers to that TiFlash finishes a Coprocessor request |


Suggested change

| [ 30 ] <Debug> CoprocessorHandler: grpc::Status DB::CoprocessorHandler::execute() | `Handle DAG request done` refers to that TiFlash finishes a Coprocessor request |

| [ 30 ] <Debug> CoprocessorHandler: grpc::Status DB::CoprocessorHandler::execute() | Handling DAG request done, that is, TiFlash finishes handling a Coprocessor request |

yikeke · 2020-04-08T12:26:22Z

reference/tiflash/maintain.md

+    LD_LIBRARY_PATH=./ ./tiflash version
+    ```
+
+- Check the TiFlash version by referring to the TiFlash log. For the log path, see the `[logger]` part in [configure the `tiflash.toml` file](/reference/tiflash/configuration.md#configure-the-tiflashtoml-file). For example:


Suggested change

- Check the TiFlash version by referring to the TiFlash log. For the log path, see the `[logger]` part in [configure the `tiflash.toml` file](/reference/tiflash/configuration.md#configure-the-tiflashtoml-file). For example:

- Check the TiFlash version by referring to the TiFlash log. For the log path, see the `[logger]` part in [the `tiflash.toml` file](/reference/tiflash/configuration.md#configure-the-tiflashtoml-file). For example:

Delete configure so the sentence is complete.

yikeke · 2020-04-08T12:36:06Z

reference/tiflash/maintain.md

+    alter table <db-name>.<table-name> set tiflash replica 0;
+    ```
+
+2. To ensure that the TiFlash replicas of these tables are removed, see [View the Table Replication Progress](/reference/tiflash/use-tiflash.md#view-the-table-replication-progress). If you cannot view the replication progress of the related tables, it means that the replicas are removed.


Suggested change

2. To ensure that the TiFlash replicas of these tables are removed, see [View the Table Replication Progress](/reference/tiflash/use-tiflash.md#view-the-table-replication-progress). If you cannot view the replication progress of the related tables, it means that the replicas are removed.

2. To ensure that the TiFlash replicas of these tables are removed, see [View the Table Replication Progress](/reference/tiflash/use-tiflash.md#view-the-table-replication-progress). If you cannot view the replication progress of the related tables, it means that the replicas are removed.

The anchor is invalid. Please confirm again. (Be careful of the links!) @toutdesuite

yikeke · 2020-04-08T12:39:58Z

reference/tiflash/maintain.md

+
+This is because TiFlash is in an abnormal state caused by configuration errors or environment issues. Take the following steps to identify the faulty component:
+
+1. Check whether PD enables the `Placement Rules` feature (to enable the feature, see the step 2 of [Add a TiFlash component in an existing TiDB Cluster](/reference/tiflash/deploy.md#add-a-TiFlash-component-in-an-existing-TiDB-cluster):


Suggested change

1. Check whether PD enables the `Placement Rules` feature (to enable the feature, see the step 2 of [Add a TiFlash component in an existing TiDB Cluster](/reference/tiflash/deploy.md#add-a-TiFlash-component-in-an-existing-TiDB-cluster):

1. Check whether PD enables the `Placement Rules` feature (to enable the feature, see the step 2 of [Add a TiFlash component in an existing TiDB Cluster](/reference/tiflash/deploy.md#add-a-TiFlash-component-in-an-existing-TiDB-cluster):

This anchor is also wrong. Please read our link style guide carefully! @toutdesuite

toutdesuite · 2020-04-08T13:30:27Z

@yikeke PTAL again

reference/tiflash/maintain.md

yikeke · 2020-04-08T14:54:45Z

/merge

sre-bot · 2020-04-08T14:55:08Z

/run-all-tests

* tiflash: add the maintain.md doc * remove dead link * modify anchor * modify anchor * modify anchor * address comments * address comments * address comments, esp anchor links * Update reference/tiflash/maintain.md * minor edits Co-authored-by: Keke Yi <40977455+yikeke@users.noreply.github.com> Co-authored-by: yikeke <yikeke@pingcap.com>

sre-bot · 2020-04-08T14:57:45Z

cherry pick to release-3.1 in PR #2213

* tiflash: add the maintain.md doc * remove dead link * modify anchor * modify anchor * modify anchor * address comments * address comments * address comments, esp anchor links * Update reference/tiflash/maintain.md * minor edits Co-authored-by: Keke Yi <40977455+yikeke@users.noreply.github.com> Co-authored-by: yikeke <yikeke@pingcap.com>

sre-bot · 2020-04-08T14:58:54Z

cherry pick to release-4.0 in PR #2214

* tiflash: add the maintain.md doc * remove dead link * modify anchor * modify anchor * modify anchor * address comments * address comments * address comments, esp anchor links * Update reference/tiflash/maintain.md * minor edits Co-authored-by: Keke Yi <40977455+yikeke@users.noreply.github.com> Co-authored-by: yikeke <yikeke@pingcap.com> Co-authored-by: toutdesuite <guizhiluo2014@163.com> Co-authored-by: Keke Yi <40977455+yikeke@users.noreply.github.com> Co-authored-by: yikeke <yikeke@pingcap.com>

tiflash: add the maintain.md doc

573a267

toutdesuite requested review from yikeke and ilovesoup April 7, 2020 12:09

toutdesuite added needs-cherry-pick-3.1 priority/P2 The issue has P2 priority. translation/from-docs-cn This PR is translated from a PR in pingcap/docs-cn. v4.0 This PR/issue applies to TiDB v4.0. labels Apr 7, 2020

toutdesuite commented Apr 7, 2020

View reviewed changes

ilovesoup approved these changes Apr 7, 2020

View reviewed changes

yikeke mentioned this pull request Apr 8, 2020

tiflash: add scale.md doc #2184

Merged

5 tasks

toutdesuite added 3 commits April 8, 2020 17:41

remove dead link

eff18a8

modify anchor

8225471

modify anchor

e7ff63c