ebs br: restore could hang if some tikv nodes are killed or restarted

[BornChanger]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Kill some TiKV node during ebs br restore phase

2. What did you expect to see? (Required)

EBS BR restore continue and succeed

3. What did you see instead (Required)

EBS BR restore hangs

4. What is your TiDB version? (Required)

TiDB 6.5 and above

[BornChanger]

/assign @YuJuncen

[YuJuncen]

This is because when we are in recovery mode, all elections will be suspended until BR choose the leader. But the problem is that AFTER BR had chosen the leader, the store got down. Once it reboots, the leaders are dropped. However we are still in recovery mode, so we cannot elect new leaders.

[YuJuncen]

A solution might be extending the recovery mode. Make it have 3 stages:

• on: the initial stage, which stops raft election and optimize for flashing back.
• for_flashback: once BR finished the wait_apply RPC, it will issue a RPC to PD that updating the recovery mode state to for_flashback, but not reboot TiKVs. That means, config of most TiKVs will be in on stage. And rebooted stores can now issue elections.
• off or unset: the default, using the unchanged config.

Once BR detected there is a TiKV outage (maybe by creating a no-op TCP connection with the gRPC port of each TiKV), BR will:

• If the current recovery mode state is on, retry the whole procedure. (This might be implemented via exit and let operator to restart it.)
• If the current recovery mode state is for_flashback, retry from flashback, and operator should reboot all stores, so all stores(If not all stores are rebooted, the rest of stores will reject voting because they believe the old leader's lease hasn't expired) will be able to electing new leaders(Perhaps we also need to resume balance-leader-scheduler at this stage.).
• If the current recovery mode state is off, do nothing and exit. (We have already successed!)

[YuJuncen]

cc @hicqu , do you have some good ideas?