Skip to content

Commit

Permalink
tools/lightning: update wording for multiple lighting docs (#1121)
Browse files Browse the repository at this point in the history
* update wording for multiple lighting docs

* Update tools/lightning/deployment.md

Co-Authored-By: Caitin <34535727+CaitinChen@users.noreply.github.com>

* Update tools/lightning/deployment.md

Co-Authored-By: Caitin <34535727+CaitinChen@users.noreply.github.com>

* remove spaces around “/”
  • Loading branch information
dcalvin authored and CaitinChen committed May 30, 2019
1 parent 3ceb2db commit 0d6cba5
Show file tree
Hide file tree
Showing 3 changed files with 49 additions and 49 deletions.
4 changes: 2 additions & 2 deletions dev/how-to/troubleshoot/tidb-lightning.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,13 +73,13 @@ See the [Checkpoints control](/dev/reference/tools/tidb-lightning/checkpoints.md
**Solutions**:
1. Increase the value of `max-open-engines` setting in `tikv-importer.toml`. This value is typically dictated by the available memory. This could be calculated as:
1. Increase the value of `max-open-engines` setting in `tikv-importer.toml`. This value is typically dictated by the available memory. This could be calculated by using:
Max Memory Usage ≈ `max-open-engines` × `write-buffer-size` × `max-write-buffer-number`
2. Decrease the value of `table-concurrency` + `index-concurrency` so it is less than `max-open-engines`.
3. Restart `tikv-importer` to forcefully remove all engine files (default to `./data.import/`). This also removes all partially imported tables, thus it is required to clear the outdated checkpoints.
3. Restart `tikv-importer` to forcefully remove all engine files (default to `./data.import/`). This also removes all partially imported tables, which requires Lightning to clear the outdated checkpoints.
```sh
tidb-lightning-ctl --config conf/tidb-lightning.toml --checkpoint-error-destroy=all
Expand Down
14 changes: 7 additions & 7 deletions dev/reference/tools/tidb-lightning/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ To achieve the best performance, it is recommended to use the following hardware

If you have sufficient machines, you can deploy multiple Lightning/Importer servers, with each working on a distinct set of tables, to import the data in parallel.

> **Notes:**
> **Note:**
>
> `tidb-lightning` is a CPU intensive program. In an environment with mixed components, the resources allocated to `tidb-lightning` must be limited. Otherwise, other components might not be able to run. It is recommended to set the `region-concurrency` to 75% of CPU logical cores. For instance, if the CPU has 32 logical cores, you can set the `region-concurrency` to 24.
> - `tidb-lightning` is a CPU intensive program. In an environment with mixed components, the resources allocated to `tidb-lightning` must be limited. Otherwise, other components might not be able to run. It is recommended to set the `region-concurrency` to 75% of CPU logical cores. For instance, if the CPU has 32 logical cores, you can set the `region-concurrency` to 24.
>
> `tikv-importer` stores intermediate data on the RAM to speed up process. The typical memory usage can be calculated from configuration as **(`max-open-engines` × `write-buffer-size` × 2) + (`num-import-jobs` × `region-split-size` × 2)**. If the speed of writing to disk is slow, the memory usage could be even higher due to buffering.
> - `tikv-importer` stores intermediate data on the RAM to speed up the import process. The typical memory usage can be calculated by using **(`max-open-engines` × `write-buffer-size` × 2) + (`num-import-jobs` × `region-split-size` × 2)**. If the speed of writing to disk is slow, the memory usage could be even higher due to buffering.

Additionally, the target TiKV cluster should have enough space to absorb the new data.
Besides [the standard requirements](/dev/how-to/deploy/hardware-recommendations.md), the total free space of the target TiKV cluster should be larger than **Size of data source × [Number of replicas](/faq/tidb.md#is-the-number-of-replicas-in-each-region-configurable-if-yes-how-to-configure-it) × 2**.
Expand Down Expand Up @@ -240,10 +240,10 @@ Download the TiDB-Lightning package (choose the same version as that of the TiDB
# Maximum number of open engines.
max-open-engines = 8
# Maximum upload speed (bytes per second) from Importer to TiKV.
#upload-speed-limit = "512MB"
# minimum ratio of target store available space: store_available_space / store_capacity.
# upload-speed-limit = "512MB"
# Minimum ratio of available space on the target store: `store_available_space`/`store_capacity`.
# Importer pauses uploading SST if the availability ratio of the target store is less than this
# value, to give PD enough time to balance regions.
# value, to allow enough time for PD to balance Regions.
min-available-ratio = 0.05
```

Expand Down Expand Up @@ -343,7 +343,7 @@ Download the TiDB-Lightning package (choose the same version as that of the TiDB
# increases the size of the first few batches to properly distribute
# resources. The scale up factor is controlled by this parameter, which
# expresses the ratio of duration between the "import" and "write" steps
# with full concurrency. This can be calculated as the ratio
# with full concurrency. This can be calculated by using the ratio
# (import duration/write duration) of a single table of size around 1 GB.
# The exact timing can be found in the log. If "import" is faster, the batch
# size variance is smaller, and a ratio of zero means a uniform batch size.
Expand Down
80 changes: 40 additions & 40 deletions dev/reference/tools/tidb-lightning/monitor.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,60 +64,60 @@ Metrics provided by `tikv-importer` are listed under the namespace `tikv_import_

- **`tikv_import_rpc_duration`** (Histogram)

Bucketed histogram of total duration needed to complete an RPC action. Labels:
Bucketed histogram for the duration of an RPC action. Labels:

- **request**: `switch_mode` / `open_engine` / `write_engine` / `close_engine` / `import_engine` / `cleanup_engine` / `compact_cluster` / `upload` / `ingest` / `compact`
- **result**: `ok` / `error`
- **request**: `switch_mode`/`open_engine`/`write_engine`/`close_engine`/`import_engine`/`cleanup_engine`/`compact_cluster`/`upload`/`ingest`/`compact`
- **result**: `ok`/`error`

- **`tikv_import_write_chunk_bytes`** (Histogram)

Bucketed histogram of the uncompressed size of a block of KV pairs received from Lightning.
Bucketed histogram for the uncompressed size of a block of KV pairs received from Lightning.

- **`tikv_import_write_chunk_duration`** (Histogram)

Bucketed histogram of the time needed to receive a block of KV pairs from Lightning.
Bucketed histogram for the time needed to receive a block of KV pairs from Lightning.

- **`tikv_import_upload_chunk_bytes`** (Histogram)

Bucketed histogram of the compressed size of a chunk of SST file uploaded to TiKV.
Bucketed histogram for the compressed size of a chunk of SST file uploaded to TiKV.

- **`tikv_import_upload_chunk_duration`** (Histogram)

Bucketed histogram of the time needed to upload a chunk of SST file to TiKV.
Bucketed histogram for the time needed to upload a chunk of SST file to TiKV.

- **`tikv_import_range_delivery_duration`** (Histogram)

Bucketed histogram of the time needed to deliver a range of KV pairs into a `dispatch-job`.
Bucketed histogram for the time needed to deliver a range of KV pairs into a `dispatch-job`.

- **`tikv_import_split_sst_duration`** (Histogram)

Bucketed histogram of the time needed to split off a range from the engine file into a single SST file.
Bucketed histogram for the time needed to split off a range from the engine file into a single SST file.

- **`tikv_import_sst_delivery_duration`** (Histogram)

Bucketed histogram of the time needed to deliver an SST file from a `dispatch-job` to an `ImportSSTJob`.
Bucketed histogram for the time needed to deliver an SST file from a `dispatch-job` to an `ImportSSTJob`.

- **`tikv_import_sst_recv_duration`** (Histogram)

Bucketed histogram of the time needed to receive an SST file from a `dispatch-job` in an `ImportSSTJob`.
Bucketed histogram for the time needed to receive an SST file from a `dispatch-job` in an `ImportSSTJob`.

- **`tikv_import_sst_upload_duration`** (Histogram)

Bucketed histogram of the time needed to upload an SST file from an `ImportSSTJob` to a TiKV node.
Bucketed histogram for the time needed to upload an SST file from an `ImportSSTJob` to a TiKV node.

- **`tikv_import_sst_chunk_bytes`** (Histogram)

Bucketed histogram of the compressed size of the SST file uploaded to a TiKV node.
Bucketed histogram for the compressed size of the SST file uploaded to a TiKV node.

- **`tikv_import_sst_ingest_duration`** (Histogram)

Bucketed histogram of the time needed to ingest an SST file into TiKV.
Bucketed histogram for the time needed to ingest an SST file into TiKV.

- **`tikv_import_each_phase`** (Gauge)

Indicates the running phase. Values can be 1, meaning running inside the phase, or 0, meaning outside the phase. Labels:
Indicates the running phase. Possible values are 1, meaning running inside the phase, and 0, meaning outside the phase. Labels:

- **phase**: `prepare` / `import`
- **phase**: `prepare`/`import`

- **`tikv_import_wait_store_available_count`** (Counter)

Expand All @@ -131,76 +131,76 @@ Metrics provided by `tidb-lightning` are listed under the namespace `lightning_*

- **`lightning_importer_engine`** (Counter)

Counting open and closed engine files. Labels:
Counts open and closed engine files. Labels:

- **type**: `open` / `closed`
- **type**: `open`/`closed`

- **`lightning_idle_workers`** (Gauge)

Counting idle workers. Values should be less than the `*-concurrency` settings and are typically zero. Labels:
Counts idle workers. Values should be less than the `*-concurrency` settings and are typically zero. Labels:

- **name**: `table` / `index` / `region` / `io` / `closed-engine`
- **name**: `table`/`index`/`region`/`io`/`closed-engine`

- **`lightning_kv_encoder`** (Counter)

Counting open and closed KV encoders. KV encoders are in-memory TiDB instances that convert SQL `INSERT` statements into KV pairs. The net values need to be bounded in a healthy situation. Labels:
Counts open and closed KV encoders. KV encoders are in-memory TiDB instances that convert SQL `INSERT` statements into KV pairs. The net values need to be bounded in a healthy situation. Labels:

- **type**: `open` / `closed`
- **type**: `open`/`closed`

* **`lightning_tables`** (Counter)

Counting number of tables processed and their status. Labels:
Counts number of tables processed and their status. Labels:

- **state**: `pending` / `written` / `closed` / `imported` / `altered_auto_inc` / `checksum` / `analyzed` / `completed`
- **result**: `success` / `failure`
- **state**: `pending`/`written`/`closed`/`imported`/`altered_auto_inc`/`checksum`/`analyzed`/`completed`
- **result**: `success`/`failure`

* **`lightning_engines`** (Counter)

Counting number of engine files processed and their status. Labels:
Counts number of engine files processed and their status. Labels:

- **state**: `pending` / `written` / `closed` / `imported` / `completed`
- **result**: `success` / `failure`
- **state**: `pending`/`written`/`closed`/`imported`/`completed`
- **result**: `success`/`failure`

- **`lightning_chunks`** (Counter)

Counting number of chunks processed and their status. Labels:
Counts number of chunks processed and their status. Labels:

- **state**: `estimated` / `pending` / `running` / `finished` / `failed`
- **state**: `estimated`/`pending`/`running`/`finished`/`failed`

- **`lightning_import_seconds`** (Histogram)

Bucketed histogram of the time needed to import a table.
Bucketed histogram for the time needed to import a table.

- **`lightning_row_read_bytes`** (Histogram)

Bucketed histogram of the size of a single SQL row.
Bucketed histogram for the size of a single SQL row.

- **`lightning_row_encode_seconds`** (Histogram)

Bucketed histogram of the time needed to encode a single SQL row into KV pairs.
Bucketed histogram for the time needed to encode a single SQL row into KV pairs.

- **`lightning_row_kv_deliver_seconds`** (Histogram)

Bucketed histogram of the time needed to deliver a set KV pairs corresponding to one single SQL row.
Bucketed histogram for the time needed to deliver a set of KV pairs corresponding to one single SQL row.

- **`lightning_block_deliver_seconds`** (Histogram)

Bucketed histogram of the time needed to deliver of a block of KV pairs to Importer.
Bucketed histogram for the time needed to deliver a block of KV pairs to Importer.

- **`lightning_block_deliver_bytes`** (Histogram)

Bucketed histogram of the uncompressed size of a block of KV pairs delivered to Importer.
Bucketed histogram for the uncompressed size of a block of KV pairs delivered to Importer.

- **`lightning_chunk_parser_read_block_seconds`** (Histogram)

Bucketed histogram of the time needed by the data file parser to read a block.
Bucketed histogram for the time needed by the data file parser to read a block.

- **`lightning_checksum_seconds`** (Histogram)

Bucketed histogram of the time taken to compute the checksum of a table.
Bucketed histogram for the time needed to compute the checksum of a table.

- **`lightning_apply_worker_seconds`** (Histogram)

Bucketed histogram of the time taken to acquire an idle worker. Labels:
Bucketed histogram for the time needed to acquire an idle worker. Labels:

- **name**: `table` / `index` / `region` / `io` / `closed-engine`
- **name**: `table`/`index`/`region`/`io`/`closed-engine`

0 comments on commit 0d6cba5

Please sign in to comment.