[UCP] Noise reduction support when cluster auto scaling #2307

vincent178 · 2020-04-26T17:48:05Z

What problem does this PR solve?

What is changed and how does it work?

Add Phrase for TidbClusterAutoScaler, it has three optional values: Normal, ReadyToScaleOut, ReadyToScaleIn.
Add user defined autoscale threshold.

use tidb as an example, tikv should be the same.

check phase
after autoscaler calculate the target replica, if target replica equals to current replica, set auto scaler phase to normal and return, otherwise check if the phase equal to scaleOut or scaleIn, if not, update the phase and record timestamp. go to 2.
check timestamp threshold
check the record timestamp remains longer than threshold which user defined. If not, return, otherwise go to 3.
do autoscale
set phase to normal and do normal autoscale.

Design for e2e tests:
After mock response from Prometheus, without noise reduction, it will start auto scaling after maximum 30s, but with noise reduction, we expect at least in 300s (ReadyToScaleThresholdSeconds),
cluster should remain the replica number. And after ReadyToScaleThresholdSeconds time, the cluster should start normal auto scaling and auto scale phase should be back to normal.

Check List

Tests

Unit test
E2E test

Code changes

Has Go code change

Side effects

n/a

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Does this PR introduce a user-facing change?:

User now can specify `ReadyToScaleThresholdSeconds` in `TidbClusterAutoScaler`, until cluster keep in ready to auto scale status for `ReadyToScaleThresholdSeconds` seconds, then cluster start auto scaling.

sre-bot · 2020-04-26T17:48:09Z

Thanks for your contribution. If your PR get merged, you will be rewarded 2000 points.

CLAassistant · 2020-04-26T17:48:12Z

All committers have signed the CLA.

Yisaer

The code seems generally good to me. Please address the comment.

And there is one more question for me about the time-window noise-reduction. It seems the tidb autoscaler should have the same result Scalein / ScaleOut in a certain time then it would auto-scale. If the Operator is down during the time-window, would the noise reduction fail?

For example:
0:01s Tidb auto-scaler have ScaleOut result, the status become ScaleOut
0:02s Operator Down
4:01s Operator Up
4:02s Tidb auto-scaler have ScaleOut result again, the auto-scaler admit to scale-out tidb.

In this case, I think the noise reduction fail

pkg/apis/pingcap/v1alpha1/tidbclusterautoscaler_types.go

pkg/label/label.go

pkg/autoscaler/autoscaler/util.go

pkg/autoscaler/autoscaler/tidb_autoscaler.go

cofyc · 2020-05-25T01:07:10Z

/run-e2e-tests

vincent178 · 2020-05-25T21:49:09Z

@cofyc @Yisaer @DanielZhangQD PTAL

Yisaer · 2020-05-26T01:15:40Z

/run-e2e-in-kind

Yisaer

Almost LGTM

Yisaer · 2020-05-26T04:54:32Z

pkg/apis/pingcap/v1alpha1/tidbclusterautoscaler_types.go

+	// If not set, the default ReadyToScaleThresholdSeconds will be set to 300.
+	// +optional
+	ReadyToScaleThresholdSeconds *int32 `json:"readyToScaleThresholdSeconds,omitempty"`


300 is about 10 times re-sync duration, I think it might be too long for a default value. What about 30 as it only require 1 re-sync duration.

Yisaer · 2020-05-26T04:57:46Z

pkg/autoscaler/autoscaler/util.go

+		if tac.Spec.TiKV.ReadyToScaleThresholdSeconds == nil {
+			tac.Spec.TiKV.ReadyToScaleThresholdSeconds = pointer.Int32Ptr(300)
+		}


300 is too large. ditto.

Yisaer · 2020-05-26T04:58:47Z

pkg/label/label.go

+	AnnTiKVReadyToScaleTimestamp = "tikv.tidb.pingcap.com/ready-to-scale-timestamp"
+
+	// AnnLastSyncingTimestamp records last sync timestamp
+	AnnLastSyncingTimestamp = "auto-scaling.tidb.pingcap.com/last-syncing-timestamp"


I think auto-scaling here is unnecessary, this annotation could be common used for each API Object.

* add AnnLastSyncingTimestamp timestamp * add ReadyToScaleThresholdSeconds in AutoScalerSpec * add AnnTiKVReadyToScaleTimestamp timestamp labels to record AutoScalerPhase * add Normal, ReadyToScaleOut and ReadyToScaleIn three AutoScalerPhase * add checkStsReadyAutoScalingTimestamp to check AutoScalerPhase timestamp, only for TiKV * add checkStsLastSyncTimestamp to check maximum thresholdSec allowed before reset phase to Normal, only for TiKV * add checkStsAutoScaling combine checkStsLastSyncTimestamp, checkStsReadyAutoScalingTimestamp and checkStsAutoScalingInterval * add unit tests * add integration e2e tests * update doc

Yisaer · 2020-05-27T03:04:57Z

/run-e2e-in-kind

cofyc · 2020-05-27T03:08:25Z

/merge

sre-bot · 2020-05-27T03:08:27Z

Your auto merge job has been accepted, waiting for:

2504

sre-bot · 2020-05-27T04:59:56Z

Team qidelongdongqiang complete task #2241 and get 2500 score, current score 3979

Yisaer · 2020-05-27T05:01:46Z

/run-cherry-picker

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot · 2020-05-27T05:02:32Z

cherry pick to release-1.1 in PR #2568

sre-bot added the contribution label Apr 26, 2020

vincent178 force-pushed the auto-scaling-noise-reduction branch 10 times, most recently from 52cc533 to de74eb8 Compare April 27, 2020 08:40

Yisaer reviewed Apr 27, 2020

View reviewed changes

vincent178 force-pushed the auto-scaling-noise-reduction branch 16 times, most recently from 5140412 to 90e23cf Compare April 29, 2020 21:38

vincent178 force-pushed the auto-scaling-noise-reduction branch from 305220a to dce8456 Compare May 25, 2020 01:04

vincent178 force-pushed the auto-scaling-noise-reduction branch 4 times, most recently from 5b69b83 to 579ad1b Compare May 25, 2020 21:48

vincent178 force-pushed the auto-scaling-noise-reduction branch from 579ad1b to 49a70b3 Compare May 26, 2020 01:11

Yisaer suggested changes May 26, 2020

View reviewed changes

Yisaer requested a review from DanielZhangQD May 26, 2020 04:59

vincent178 force-pushed the auto-scaling-noise-reduction branch 2 times, most recently from e3e2a7a to f620f6e Compare May 26, 2020 22:06

vincent178 force-pushed the auto-scaling-noise-reduction branch from f620f6e to 2747929 Compare May 27, 2020 02:47

Yisaer approved these changes May 27, 2020

View reviewed changes

DanielZhangQD requested a review from weekface May 27, 2020 03:04

cofyc approved these changes May 27, 2020

View reviewed changes

sre-bot added the CanMerge label May 27, 2020

Yisaer merged commit d7af535 into pingcap:master May 27, 2020

sre-bot pushed a commit to sre-bot/tidb-operator that referenced this pull request May 27, 2020

cherry pick pingcap#2307 to release-1.1

891c823

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot mentioned this pull request May 27, 2020

[UCP] Noise reduction support when cluster auto scaling (#2307) #2568

Merged

vincent178 deleted the auto-scaling-noise-reduction branch May 27, 2020 08:49

sre-bot added a commit that referenced this pull request May 29, 2020

[UCP] Noise reduction support when cluster auto scaling (#2307) (#2568)

4090243

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UCP] Noise reduction support when cluster auto scaling #2307

[UCP] Noise reduction support when cluster auto scaling #2307

vincent178 commented Apr 26, 2020 •

edited

Loading

sre-bot commented Apr 26, 2020

CLAassistant commented Apr 26, 2020 •

edited

Loading

Yisaer left a comment

cofyc commented May 25, 2020

vincent178 commented May 25, 2020

Yisaer commented May 26, 2020 •

edited

Loading

Yisaer left a comment

Yisaer May 26, 2020 •

edited

Loading

Yisaer May 26, 2020

Yisaer May 26, 2020

Yisaer commented May 27, 2020

cofyc commented May 27, 2020

sre-bot commented May 27, 2020

sre-bot commented May 27, 2020

Yisaer commented May 27, 2020

sre-bot commented May 27, 2020

[UCP] Noise reduction support when cluster auto scaling #2307

[UCP] Noise reduction support when cluster auto scaling #2307

Conversation

vincent178 commented Apr 26, 2020 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Does this PR introduce a user-facing change?:

sre-bot commented Apr 26, 2020

CLAassistant commented Apr 26, 2020 • edited Loading

Yisaer left a comment

Choose a reason for hiding this comment

cofyc commented May 25, 2020

vincent178 commented May 25, 2020

Yisaer commented May 26, 2020 • edited Loading

Yisaer left a comment

Choose a reason for hiding this comment

Yisaer May 26, 2020 • edited Loading

Choose a reason for hiding this comment

Yisaer May 26, 2020

Choose a reason for hiding this comment

Yisaer May 26, 2020

Choose a reason for hiding this comment

Yisaer commented May 27, 2020

cofyc commented May 27, 2020

sre-bot commented May 27, 2020

sre-bot commented May 27, 2020

Yisaer commented May 27, 2020

sre-bot commented May 27, 2020

vincent178 commented Apr 26, 2020 •

edited

Loading

CLAassistant commented Apr 26, 2020 •

edited

Loading

Yisaer commented May 26, 2020 •

edited

Loading

Yisaer May 26, 2020 •

edited

Loading