br: after br restore, tikv used storage is not balance

[mayjiang0203]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

run oltp_fun_001

2. What did you expect to see? (Required)

Restore finished at 18:26. Tikv used should be balance in all nodes.

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

/ # /br -V
Release Version: v5.4.0-nightly
Git Commit Hash: 76aae0d
Git Branch: heads/refs/tags/v5.4.0-nightly
Go Version: go1.16.4
UTC Build Time: 2021-12-26 08:07:37
Race Enabled: false

/ # /tidb-server -V
Release Version: v5.4.0-nightly
Edition: Community
Git Commit Hash: 76aae0d
Git Branch: heads/refs/tags/v5.4.0-nightly
UTC Build Time: 2021-12-26 08:09:11
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

Logs and monitor can be get from minio using following testbed name.
endless-oltp--tps-542284-1-875

[mayjiang0203]

Seems it is hard to reproduce it. Didn't occur after try 4 times.

[YuJuncen]

I've checked some of metrics. There are some strange things:

1. PD failed to create operator because of unhealthy regions. This can be retried after we using the new ScatterRegions API.
2. Seems BR sometimes scatters non-empty region. (Even every region are empty, the 'pending' status still exists.)

[mayjiang0203]

It occur again, seems still caused by scatter failure.

[mayjiang0203]

Hit it again， and it take user 30 min to balance after br import, so mark it Major.

[YuJuncen]

Note that v5.4 contains #30899, which can probably allay this problem.

#31691 would fix this problem.

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.