*: add br tool doc and refine mydumper/loader doc #1680

TomShawn · 2019-11-29T03:58:16Z

What is changed, added or deleted?

Add the BR tool document and refine mydumper/loader document.

What is the related PR or file link(s)?

pingcap/docs-cn#2020
pingcap/docs-cn#2088
pingcap/docs-cn#2084
pingcap/docs-cn#2089

Which version does your change affect?

dev, v3.1

dev/how-to/maintain/backup-and-restore/br.md

lilin90 · 2019-12-06T07:51:33Z

dev/how-to/maintain/backup-and-restore/br.md

+* `br restore`: used to restore the data of the TiDB cluster.
+* `br version`: used to check the version of BR.
+
+Each of the above three sub-commands might include the following three sub-commands to specify the scope of an operation:


Suggested change

Each of the above three sub-commands might include the following three sub-commands to specify the scope of an operation:

Each of the above three sub-commands might still include the following three sub-commands to specify the scope of an operation:

lilin90 · 2019-12-06T07:57:07Z

dev/how-to/maintain/backup-and-restore/br.md

+Explanations for some options in the above command are as follows:
+
+* `--ratelimit`: specifies the maximum speed at which a backup operation is performed (MiB/s) on each TiKV node.
+* `--concurrency`: sets an upper limit on the number of concurrent executions on each TiKV node.


Suggested change

* `--concurrency`: sets an upper limit on the number of concurrent executions on each TiKV node.

* `--concurrency`: sets an upper limit on the number of concurrent operations on each TiKV node.

lilin90 · 2019-12-06T07:57:52Z

dev/how-to/maintain/backup-and-restore/br.md

+
+* `--ratelimit`: specifies the maximum speed at which a backup operation is performed (MiB/s) on each TiKV node.
+* `--concurrency`: sets an upper limit on the number of concurrent executions on each TiKV node.
+* `--log-file`: specifies that the BR log is written to the `backupfull.log` file.


Suggested change

* `--log-file`: specifies that the BR log is written to the `backupfull.log` file.

* `--log-file`: specifies writing the BR log to the `backupfull.log` file.

lilin90 · 2019-12-06T07:58:40Z

dev/how-to/maintain/backup-and-restore/br.md

+* `--concurrency`: sets an upper limit on the number of concurrent executions on each TiKV node.
+* `--log-file`: specifies that the BR log is written to the `backupfull.log` file.
+
+A progress bar is displayed in the terminal during the backup. When the progress bar advances to 100%, the backup is complete. Then the BR also checks the backup data to ensure data security. The progress bar is displayed as follows:


Suggested change

A progress bar is displayed in the terminal during the backup. When the progress bar advances to 100%, the backup is complete. Then the BR also checks the backup data to ensure data security. The progress bar is displayed as follows:

A progress bar is displayed in the terminal during the backup. When the progress bar advances to 100%, the backup is complete. Then the BR also checks the backup data to ensure data safety. The progress bar is displayed as follows:

lilin90 · 2019-12-06T08:00:04Z

dev/how-to/maintain/backup-and-restore/br.md

+Full Backup <---------/................................................> 17.12%.
+```
+
+### Back up single table


Suggested change

### Back up single table

### Back up data of a single table

lilin90 · 2019-12-06T08:38:57Z

dev/how-to/maintain/backup-and-restore/br.md

+
+This section shows how to back up and restore the data of an existing cluster. You can estimate the performance of backup and restoration based on machine performance, configuration and data size.
+
+### Data size and machine configuration


Suggested change

### Data size and machine configuration

### Data volume and machine configuration

lilin90 · 2019-12-06T08:39:15Z

dev/how-to/maintain/backup-and-restore/br.md

+
+### Data size and machine configuration
+
+Suppose that the backup and restoration operations are performed on 10 tables in the TiKV cluster, each table with 5 million rows of data. The total data size is 35 GB.


Suggested change

Suppose that the backup and restoration operations are performed on 10 tables in the TiKV cluster, each table with 5 million rows of data. The total data size is 35 GB.

Suppose that the backup and restoration operations are performed on 10 tables in the TiKV cluster, each table with 5 million rows of data. The total data volume is 35 GB.

lilin90 · 2019-12-06T08:41:54Z

dev/how-to/maintain/backup-and-restore/br.md

+
+### Backup
+
+Before the backup operation, make sure the following things are done:


Suggested change

Before the backup operation, make sure the following things are done:

Before the backup operation, check the following two items:

lilin90 · 2019-12-06T08:42:22Z

dev/how-to/maintain/backup-and-restore/br.md

+Before the backup operation, make sure the following things are done:
+
+- `tikv_gc_life_time` is set to a larger value so that the backup operation will not be interrupted by data loss.
+- No DDL statement is executed on TiDB.


Suggested change

- No DDL statement is executed on TiDB.

- No DDL statement is being executed on the TiDB cluster.

lilin90 · 2019-12-06T08:43:41Z

dev/how-to/maintain/backup-and-restore/br.md

+
+Before the backup operation, make sure the following things are done:
+
+- `tikv_gc_life_time` is set to a larger value so that the backup operation will not be interrupted by data loss.


Suggested change

- `tikv_gc_life_time` is set to a larger value so that the backup operation will not be interrupted by data loss.

- You have set `tikv_gc_life_time` to a larger value so that the backup operation will not be interrupted because of data loss.

lilin90 · 2019-12-06T08:53:07Z

dev/how-to/maintain/backup-and-restore/br.md

+[INFO] [client.go:435] ["Restore Checksum"] [take=6.385818026s]
+```
+
+## Working principles


Suggested change

## Working principles

## Implementation principles

lilin90 · 2019-12-06T08:54:10Z

dev/how-to/maintain/backup-and-restore/br.md

+
+# Use BR to Backup and Restore Cluster Data
+
+Backup & Restore (BR) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with [`mydumper`/`loader`](/dev/how-to/maintain/backup-and-restore/mydumper-loader.md), BR is more suitable for scenarios of huge data volume. This document describes the BR command line, detailed use examples, best practices, restrictions, and introduces the working principles of BR.


Suggested change

Backup & Restore (BR) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with [`mydumper`/`loader`](/dev/how-to/maintain/backup-and-restore/mydumper-loader.md), BR is more suitable for scenarios of huge data volume. This document describes the BR command line, detailed use examples, best practices, restrictions, and introduces the working principles of BR.

Backup & Restore (BR) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with [`mydumper`/`loader`](/dev/how-to/maintain/backup-and-restore/mydumper-loader.md), BR is more suitable for scenarios of huge data volume. This document describes the BR command line, detailed use examples, best practices, restrictions, and introduces the implementation principles of BR.

lilin90 · 2019-12-06T08:55:33Z

dev/how-to/maintain/backup-and-restore/br.md

+
+## Working principles
+
+BR sends the backup and restoration commands to each TiKV node. After receiving these commands, TiKV performs the corresponding backup and restoration operations. Each TiKV node has a path in which the backup files generated in the backup operation are stored and from which the stored backup files are read in the restoration.


Suggested change

BR sends the backup and restoration commands to each TiKV node. After receiving these commands, TiKV performs the corresponding backup and restoration operations. Each TiKV node has a path in which the backup files generated in the backup operation are stored and from which the stored backup files are read in the restoration.

BR sends the backup and restoration commands to each TiKV node. After receiving these commands, TiKV performs the corresponding backup and restoration operations. Each TiKV node has a path in which the backup files generated in the backup operation are stored and from which the stored backup files are read during restoration.

lilin90 · 2019-12-06T08:56:42Z

dev/how-to/maintain/backup-and-restore/br.md

+
+According to these information, BR starts a TiDB instance internally to obtain the database or table information corresponding to the TS, and filters out the system databases (`information_schema`, `performance_schema`, `mysql`) at the same time.
+
+According to the backup sub-command, two types of backup logic are available:


Suggested change

According to the backup sub-command, two types of backup logic are available:

According to the backup sub-command, BR adopts the following two types of backup logic:

lilin90 · 2019-12-06T08:57:20Z

dev/how-to/maintain/backup-and-restore/br.md

+
+According to the backup sub-command, two types of backup logic are available:
+
+- Full backup: BR traverses all the tables and constructs the KV range to be backed up according to every table.


Suggested change

- Full backup: BR traverses all the tables and constructs the KV range to be backed up according to every table.

- Full backup: BR traverses all the tables and constructs the KV range to be backed up according to each table.

lilin90 · 2019-12-06T09:06:45Z

dev/how-to/maintain/backup-and-restore/br.md

+
+After TiKV receives the request to load the SST file, TiKV uses the Raft mechanism to ensure the strong consistency of the SST data. After the downloaded SST file is loaded successfully, the file is deleted asynchronously.
+
+After the restoration operation, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.


Suggested change

After the restoration operation, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.

After the restoration operation is completed, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.

lilin90 · 2019-12-06T09:08:51Z

dev/how-to/maintain/backup-and-restore/mydumper-loader.md

@@ -1,14 +1,15 @@
 ---
-title: Backup and Restore
-summary: Learn how to back up and restore the data of TiDB.
+title: Use `mydumper`/`loader` to Backup and Restore


Suggested change

title: Use `mydumper`/`loader` to Backup and Restore

title: Use `mydumper` and `loader` to Back up and Restore Data

lilin90 · 2019-12-06T09:08:59Z

dev/how-to/maintain/backup-and-restore/mydumper-loader.md

-title: Backup and Restore
-summary: Learn how to back up and restore the data of TiDB.
+title: Use `mydumper`/`loader` to Backup and Restore
+summary: Learn how to backup and restore the data of TiDB using `mydumper`/`loader`.


Suggested change

summary: Learn how to backup and restore the data of TiDB using `mydumper`/`loader`.

summary: Learn how to back up and restore the data of TiDB using `mydumper` and `loader`.

lilin90 · 2019-12-06T09:09:23Z

dev/how-to/maintain/backup-and-restore/mydumper-loader.md

 ---

-# Backup and Restore
+# Use `mydumper`/`loader` to Backup and Restore


Suggested change

# Use `mydumper`/`loader` to Backup and Restore

# Use `mydumper` and `loader` to Back up and Restore Data

lilin90 · 2019-12-06T09:09:41Z

dev/how-to/maintain/backup-and-restore/mydumper-loader.md


-This document describes how to back up and restore the data of TiDB. Currently, this document only covers full backup and restoration.
+This document describes how to back up and restore the data of TiDB using `mydumper`/`loader`. Currently, this document only covers full backup and restoration.


Suggested change

This document describes how to back up and restore the data of TiDB using `mydumper`/`loader`. Currently, this document only covers full backup and restoration.

This document describes how to back up and restore the data of TiDB using `mydumper` and `loader`. Currently, this document only covers full backup and restoration.

lilin90 · 2019-12-06T09:42:29Z

dev/how-to/maintain/backup-and-restore/br.md

+
+After TiKV receives the request to load the SST file, TiKV uses the Raft mechanism to ensure the strong consistency of the SST data. After the downloaded SST file is loaded successfully, the file is deleted asynchronously.
+
+After the restoration operation is complete, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.


Suggested change

After the restoration operation is complete, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.

After the restoration operation is completed, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.

lilin90 · 2019-12-06T09:42:38Z

v3.1/how-to/maintain/backup-and-restore/br.md

+
+After TiKV receives the request to load the SST file, TiKV uses the Raft mechanism to ensure the strong consistency of the SST data. After the downloaded SST file is loaded successfully, the file is deleted asynchronously.
+
+After the restoration operation is complete, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.


Suggested change

After the restoration operation is complete, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.

After the restoration operation is completed, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.

lilin90

LGTM

@overvenus @3pointer PTAL~ Thanks!

tools: add br tool doc and refine mydumper/loader doc

ded10fb

TomShawn added dev translation/from-docs-cn This PR is translated from a PR in pingcap/docs-cn. size/large Changes of a large size. labels Nov 29, 2019

TomShawn and others added 4 commits November 29, 2019 12:02

Merge branch 'master' into add-br-tool

89cd5dd

tools: add diagram

c4b9c04

tools: adjust image size

cd1ef1b

tools: refine language

b3b8d7b

TomShawn marked this pull request as ready for review November 29, 2019 07:54

TomShawn requested a review from overvenus November 29, 2019 08:08

tools: refine language

ce5e4f1

TomShawn requested a review from lilin90 November 29, 2019 10:15

TomShawn changed the title ~~tools: add br tool doc and refine mydumper/loader doc~~ [WIP]tools: add br tool doc and refine mydumper/loader doc Nov 29, 2019

TomShawn changed the title ~~[WIP]tools: add br tool doc and refine mydumper/loader doc~~ [WIP] tools: add br tool doc and refine mydumper/loader doc Nov 29, 2019

TomShawn and others added 3 commits December 2, 2019 19:48

maintain: update content

ba15be8

Merge branch 'master' into add-br-tool

9994d39

maintain: refine language

0805555

TomShawn changed the title ~~[WIP] tools: add br tool doc and refine mydumper/loader doc~~ maintain: add br tool doc and refine mydumper/loader doc Dec 2, 2019

maintain: fix a typo

2c4dc01

lilin90 mentioned this pull request Dec 3, 2019

backup-and-restore: add basic introduction pingcap/docs-cn#2089

Merged

lilin90 added the priority/P1 The issue has P1 priority. label Dec 3, 2019

maintain: add working principles

ea9bce0

TomShawn requested a review from 3pointer December 3, 2019 12:23

TomShawn and others added 6 commits December 3, 2019 20:34

maintain: refine introduction

3a0a9e0

maintain: refine format and layout

e93a47f

maintain: address comments from pingcap#2088

a27a1c7

Merge branch 'master' into add-br-tool

85464b8

maintainL refine language

852c40e

maintain: minor language change

e2a6c5b

Merge branch 'master' into add-br-tool

ea928ba

lilin90 reviewed Dec 6, 2019

View reviewed changes

dev/how-to/maintain/backup-and-restore/br.md Outdated Show resolved Hide resolved

dev/how-to/maintain/backup-and-restore/br.md Outdated Show resolved Hide resolved

address comments

c6ab9e4

lilin90 reviewed Dec 6, 2019

View reviewed changes

address comments

7c06d38

lilin90 reviewed Dec 6, 2019

View reviewed changes

lilin90 changed the title ~~maintain: add br tool doc and refine mydumper/loader doc~~ *: add br tool doc and refine mydumper/loader doc Dec 6, 2019

TomShawn added 2 commits December 6, 2019 17:36

address comments

9877d87

maintain: fix a typo

0d962a2

lilin90 reviewed Dec 6, 2019

View reviewed changes

maintain: fix grammar error

ce9bb44

lilin90 reviewed Dec 6, 2019

View reviewed changes

maintain: fix a typo

d9ae454

overvenus approved these changes Dec 6, 2019

View reviewed changes

lilin90 merged commit f36b733 into pingcap:master Dec 6, 2019

TomShawn deleted the add-br-tool branch December 6, 2019 12:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: add br tool doc and refine mydumper/loader doc #1680

*: add br tool doc and refine mydumper/loader doc #1680

TomShawn commented Nov 29, 2019 •

edited

Loading

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 Dec 6, 2019

lilin90 left a comment

	Each of the above three sub-commands might include the following three sub-commands to specify the scope of an operation:
	Each of the above three sub-commands might still include the following three sub-commands to specify the scope of an operation:

	* `--concurrency`: sets an upper limit on the number of concurrent executions on each TiKV node.
	* `--concurrency`: sets an upper limit on the number of concurrent operations on each TiKV node.

	* `--log-file`: specifies that the BR log is written to the `backupfull.log` file.
	* `--log-file`: specifies writing the BR log to the `backupfull.log` file.

	A progress bar is displayed in the terminal during the backup. When the progress bar advances to 100%, the backup is complete. Then the BR also checks the backup data to ensure data security. The progress bar is displayed as follows:
	A progress bar is displayed in the terminal during the backup. When the progress bar advances to 100%, the backup is complete. Then the BR also checks the backup data to ensure data safety. The progress bar is displayed as follows:


		This section shows how to back up and restore the data of an existing cluster. You can estimate the performance of backup and restoration based on machine performance, configuration and data size.

		### Data size and machine configuration

	### Data size and machine configuration
	### Data volume and machine configuration


		### Data size and machine configuration

		Suppose that the backup and restoration operations are performed on 10 tables in the TiKV cluster, each table with 5 million rows of data. The total data size is 35 GB.


		### Backup

		Before the backup operation, make sure the following things are done:

	Before the backup operation, make sure the following things are done:
	Before the backup operation, check the following two items:

	- No DDL statement is executed on TiDB.
	- No DDL statement is being executed on the TiDB cluster.


		Before the backup operation, make sure the following things are done:

		- `tikv_gc_life_time` is set to a larger value so that the backup operation will not be interrupted by data loss.

	- `tikv_gc_life_time` is set to a larger value so that the backup operation will not be interrupted by data loss.
	- You have set `tikv_gc_life_time` to a larger value so that the backup operation will not be interrupted because of data loss.


		# Use BR to Backup and Restore Cluster Data

		Backup & Restore (BR) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with [`mydumper`/`loader`](/dev/how-to/maintain/backup-and-restore/mydumper-loader.md), BR is more suitable for scenarios of huge data volume. This document describes the BR command line, detailed use examples, best practices, restrictions, and introduces the working principles of BR.


		## Working principles

		BR sends the backup and restoration commands to each TiKV node. After receiving these commands, TiKV performs the corresponding backup and restoration operations. Each TiKV node has a path in which the backup files generated in the backup operation are stored and from which the stored backup files are read in the restoration.


		According to these information, BR starts a TiDB instance internally to obtain the database or table information corresponding to the TS, and filters out the system databases (`information_schema`, `performance_schema`, `mysql`) at the same time.

		According to the backup sub-command, two types of backup logic are available:

	According to the backup sub-command, two types of backup logic are available:
	According to the backup sub-command, BR adopts the following two types of backup logic:


		According to the backup sub-command, two types of backup logic are available:

		- Full backup: BR traverses all the tables and constructs the KV range to be backed up according to every table.


		After TiKV receives the request to load the SST file, TiKV uses the Raft mechanism to ensure the strong consistency of the SST data. After the downloaded SST file is loaded successfully, the file is deleted asynchronously.

		After the restoration operation, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.

	After the restoration operation, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.
	After the restoration operation is completed, BR performs a checksum calculation on the restored data to compare the stored data with the backed up data.

	title: Use `mydumper`/`loader` to Backup and Restore
	title: Use `mydumper` and `loader` to Back up and Restore Data

*: add br tool doc and refine mydumper/loader doc #1680

*: add br tool doc and refine mydumper/loader doc #1680

Conversation

TomShawn commented Nov 29, 2019 • edited Loading

What is changed, added or deleted?

What is the related PR or file link(s)?

Which version does your change affect?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilin90 left a comment

Choose a reason for hiding this comment

TomShawn commented Nov 29, 2019 •

edited

Loading

	summary: Learn how to backup and restore the data of TiDB using `mydumper`/`loader`.
	summary: Learn how to back up and restore the data of TiDB using `mydumper` and `loader`.

	# Use `mydumper`/`loader` to Backup and Restore
	# Use `mydumper` and `loader` to Back up and Restore Data


		This document describes how to back up and restore the data of TiDB. Currently, this document only covers full backup and restoration.
		This document describes how to back up and restore the data of TiDB using `mydumper`/`loader`. Currently, this document only covers full backup and restoration.