[UCP] Noise reduction support when cluster auto scaling (#2307)

[sre-bot]

cherry-pick #2307 to release-1.1

---

What problem does this PR solve?

UCP #2241

What is changed and how does it work?

Add Phrase for TidbClusterAutoScaler, it has three optional values: Normal, ReadyToScaleOut, ReadyToScaleIn.
Add user defined autoscale threshold.

use tidb as an example, tikv should be the same.

1. check phase
   after autoscaler calculate the target replica, if target replica equals to current replica, set auto scaler phase to normal and return, otherwise check if the phase equal to scaleOut or scaleIn, if not, update the phase and record timestamp. go to 2.

2. check timestamp threshold
   check the record timestamp remains longer than threshold which user defined. If not, return, otherwise go to 3.

3. do autoscale
   set phase to normal and do normal autoscale.

Design for e2e tests:
After mock response from Prometheus, without noise reduction, it will start auto scaling after maximum 30s, but with noise reduction, we expect at least in 300s (ReadyToScaleThresholdSeconds),
cluster should remain the replica number. And after ReadyToScaleThresholdSeconds time, the cluster should start normal auto scaling and auto scale phase should be back to normal.

Check List

Tests

• Unit test
• E2E test

Code changes

• Has Go code change

Side effects

n/a

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Does this PR introduce a user-facing change?:

User now can specify `ReadyToScaleThresholdSeconds` in `TidbClusterAutoScaler`, until cluster keep in ready to auto scale status for `ReadyToScaleThresholdSeconds` seconds, then cluster start auto scaling.

[DanielZhangQD]

/merge

[sre-bot]

/run-all-tests

[sre-bot]

@sre-bot merge failed.

[Yisaer]

We would bring this to the minor version after 1.1 GA releases.

[Yisaer]

/merge

[sre-bot]

/run-all-tests