backend: split and ingest region size more precise

[glorv]

What problem does this PR solve?

try to resolve #365 by avoid writing and ingesting big regions

What is changed and how it works?

make ingest region size not exceed regionSplitSize by:

• use sample data to calculate kv range for each region.
• Sample. When split range, first split the range from start-key to end-key to (total kv count) / 100 ranges. Then seek to the key that is greater or equal to each range start. If the seeked key is smaller than the range end, add the range end to sample values. If there is enough sampled values, then split the sample values to the target count and each split end with be the range end. If there are no enough sample values, we will fall back to the old naive way to split, but will use the db iterator the check empty ranges and skip them.
• Region Size. when split regions, use 3/4 of the splitRegionSize to calculate the split size. And when ingest, use 5/4 splitRegionSize as the threshold. At each write and ingest phase, write no more than this threshold byte to tikv and ingest it. If there are remian kvs, split target region and try to ingest the remain bytes. By this way, we can make sure each region size will not exceed 5/4 * splitRegionSize in bytes.

This pr also emit some useless logs to make the error and warn logs less annoying.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Possible performance regression
  • benchmark with 14k warehouse tpcc data shows no performance regression
• Increased code complexity

Related changes

[glorv]

sad for the code coverage decrease, the new logic for split need large scale test data

[breezewish]

I'm not sure what's the implementation in this PR, but a kind reminder: In TiKV there are two region split methods, by size and by number of keys. When size or number of keys exceed the threshold the region will be splited. In lightning we'd better respect both mechanism, to avoid too many keys (which will also slow down the scanning).

[glorv]

I'm not sure what's the implementation in this PR, but a kind reminder: In TiKV there are two region split methods, by size and by number of keys. When size or number of keys exceed the threshold the region will be splited. In lightning we'd better respect both mechanism, to avoid too many keys (which will also slow down the scanning).

Thanks for the reminder. I update the split regions by take both size in bytes and key-value count into account.

[glorv]

/run-all-tests

[glorv]

@kennytm @3pointer @YuJuncen PTAL again

[3pointer]

Rest LGTM

[kennytm]

LGTM

[3pointer]

LGTM