[store/tikv] avoid too many times of backoff when we retry for batch cop

[hanfei1991]

What problem does this PR solve?

Problem Summary:

When a store has failed and we retry a batch cop request, we apply backoff for every region, which results in too long wait-time. Indeed we should backoff once for every store fail.

What is changed and how it works?

we should backoff once for every store fail.

What's Changed:

How it Works:

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch

Check List

Tests

• Manual test
  • we simulate the batch cop request which many regions and a store crashes. After change, tidb can switch to another store rapidly.

Side effects

None

Release note

• avoid too many times of backoff when we retry for batch cop

[codecov]

Codecov Report

Merging #18999 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #18999   +/-   ##
===========================================
  Coverage   79.3846%   79.3846%           
===========================================
  Files           546        546           
  Lines        147943     147943           
===========================================
  Hits         117444     117444           
  Misses        21007      21007           
  Partials       9492       9492           

[JaySon-Huang]

Normally run select count(*) from tpch_1.lineitem takes about 0.06 seconds.
Before this PR, after some TiFlash stores are down, with tidb_allow_batch_cop=1, running count(*) takes about a few seconds to tens of seconds. While with tidb_allow_batch_cop=0, it only takes about 0.1x seconds.
After this PR, there should be no significant difference no matter super batch is enabled or not.

[lysu]

LGTM

[lzmhhh123]

LGTM

[lzmhhh123]

/merge

[ti-srebot]

/run-all-tests

[ti-srebot]

cherry pick to release-4.0 in PR #19017