analyze: analyze table after restore #605

3pointer · 2020-11-20T05:55:49Z

What problem does this PR solve?

close #598

What is changed and how it works?

backup stats json in backup meta.
restore stats after validate checksum.

Check List

Tests

Manual test (add detailed scripts or steps below)

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Release Note

Fix the issue that didn't analyze table after restoration.

3pointer · 2020-11-25T05:04:38Z

/rebuild

zhouqiang-cl · 2020-11-25T05:24:45Z

/rebuild

3pointer · 2020-11-25T06:01:02Z

/rebuild

3pointer · 2020-11-25T06:04:51Z

/rebuild

pkg/backup/client.go

kennytm · 2020-11-25T08:18:32Z

cmd/backup.go

@@ -56,8 +55,6 @@ func NewBackupCommand() *cobra.Command {

 			// Do not run ddl worker in BR.
 			ddl.RunWorker = false
-			// Do not run stat worker in BR.
-			session.DisableStats4Test()


can we disable the stats loop after we have dumped all JSONTables?

when we reached here, we have a domain already. so it seems we can not touch the statsLease any more, https://github.com/pingcap/tidb/blob/e136429d8dc5d70f43cd3f94179b0b9f47595097/session/tidb.go#L68

Co-authored-by: kennytm <kennytm@gmail.com>

3pointer · 2020-11-25T08:50:50Z

/rebuild

glorv · 2020-11-25T09:05:04Z

/run-all-tests

glorv · 2020-11-25T09:05:29Z

/build

kennytm

please add a test involving restoring from an older archive where the stats are missing (nil).

(perhaps add a flag like br backup --stats=false too to simplify testing)

pkg/restore/batcher_test.go

3pointer · 2020-11-25T12:27:26Z

/rebuild

glorv · 2020-11-25T12:52:35Z

pkg/restore/batcher_test.go

 	for _, t := range tables {
 		log.Info("entering", zap.Int64("table ID", t.Table.ID))
-		manager[t.Table.ID] = true
+		manager.lock.Lock()


move this lock outsize the for loop

glorv · 2020-11-25T12:52:45Z

pkg/restore/batcher_test.go

 	for _, t := range tables {
-		if !manager[t.Table.ID] {
+		manager.lock.Lock()


glorv · 2020-11-25T12:53:35Z

pkg/restore/batcher_test.go

 	for _, t := range tables {
 		ids = append(ids, t.Table.ID)
 	}
-	for id, contains := range manager {
+	for id, contains := range manager.m {


Read option should be protected by lock

3pointer · 2020-11-25T13:27:04Z

please add a test involving restoring from an older archive where the stats are missing (nil).

(perhaps add a flag like br backup --stats=false too to simplify testing)

Done， PTAL

pkg/task/backup.go

lichunzhu

LGTM

3pointer · 2020-11-26T07:02:55Z

/rebuild

glorv · 2020-11-26T07:05:20Z

/build

glorv · 2020-11-26T07:08:32Z

/build

kennytm

rest LGTM

pkg/restore/batcher_test.go

pkg/restore/client.go

kennytm · 2020-11-26T08:33:23Z

LGTM

glorv · 2020-11-26T08:53:01Z

/merge

ti-srebot · 2020-11-26T08:53:18Z

/run-all-tests

ti-srebot · 2020-11-26T09:05:18Z

cherry pick to release-4.0 failed

* cherry-pick #610 * cherry-pick #605 * fix: backup will skip empty databases. (#560) Co-authored-by: 3pointer <luancheng@pingcap.com> Co-authored-by: tangwz <tangwz.com@gmail.com> Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com> Co-authored-by: kennytm <kennytm@gmail.com>

3pointer added 4 commits November 20, 2020 13:52

analyze: analyze table after restore

02c56d7

update kvproto & add test

6c65aca

fix build

29922bc

fix go mod

d18a920

3pointer added the needs-cherry-pick-release-4.0 label Nov 24, 2020

3pointer added this to the v4.0.9 milestone Nov 24, 2020

3pointer added 3 commits November 25, 2020 11:18

update tidb to complie

979f921

Merge branch 'master' into analyze

10cc69e

fix test

eeb34ec

3pointer marked this pull request as ready for review November 25, 2020 04:49

3pointer requested review from overvenus, glorv and kennytm November 25, 2020 04:50

3pointer added 3 commits November 25, 2020 13:27

fix test

ce079e4

fix build

53640f0

fix ci

2d5280b

kennytm reviewed Nov 25, 2020

View reviewed changes

3pointer and others added 2 commits November 25, 2020 16:37

Update pkg/backup/client.go

93dcb1a

Co-authored-by: kennytm <kennytm@gmail.com>

Merge branch 'master' into analyze

a83cd17

fix ci failed

90f2340

kennytm reviewed Nov 25, 2020

View reviewed changes

lichunzhu reviewed Nov 25, 2020

View reviewed changes

pkg/restore/batcher_test.go Show resolved Hide resolved

address comment

8475315

3pointer added 3 commits November 25, 2020 18:50

address comment

a2ccf2b

fix ci

64db129

add test for old backup

9c91339

update comments

80be6d4

glorv reviewed Nov 25, 2020

View reviewed changes

address comment

8c986b2

lichunzhu reviewed Nov 26, 2020

View reviewed changes

pkg/task/backup.go Outdated Show resolved Hide resolved

address comment

5440134

lichunzhu reviewed Nov 26, 2020

View reviewed changes

ti-srebot added the status/LGT1 LGTM1 label Nov 26, 2020

kennytm reviewed Nov 26, 2020

View reviewed changes

pkg/restore/batcher_test.go Show resolved Hide resolved

pkg/restore/batcher_test.go Show resolved Hide resolved

pkg/restore/client.go Outdated Show resolved Hide resolved

3pointer added 2 commits November 26, 2020 16:27

address comment

5a00a7b

Merge branch 'master' into analyze

e8a513a

ti-srebot removed the status/LGT1 LGTM1 label Nov 26, 2020

ti-srebot approved these changes Nov 26, 2020

View reviewed changes

ti-srebot added the status/LGT2 LGTM2 label Nov 26, 2020

ti-srebot added the CanMerge label Nov 26, 2020

ti-srebot merged commit 1a7a2f3 into pingcap:master Nov 26, 2020

lichunzhu pushed a commit to lichunzhu/br that referenced this pull request Nov 26, 2020

cherry-pick pingcap#605

f09958b

overvenus mentioned this pull request Dec 10, 2020

Dump and load statistics #429

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

analyze: analyze table after restore #605

analyze: analyze table after restore #605

3pointer commented Nov 20, 2020

3pointer commented Nov 25, 2020

zhouqiang-cl commented Nov 25, 2020

3pointer commented Nov 25, 2020

3pointer commented Nov 25, 2020

kennytm Nov 25, 2020

3pointer Nov 25, 2020 •

edited

Loading

3pointer commented Nov 25, 2020

glorv commented Nov 25, 2020

glorv commented Nov 25, 2020

kennytm left a comment

3pointer commented Nov 25, 2020

glorv Nov 25, 2020

glorv Nov 25, 2020

glorv Nov 25, 2020

3pointer commented Nov 25, 2020

lichunzhu left a comment

3pointer commented Nov 26, 2020

glorv commented Nov 26, 2020

glorv commented Nov 26, 2020

kennytm left a comment

kennytm commented Nov 26, 2020

glorv commented Nov 26, 2020

ti-srebot commented Nov 26, 2020

ti-srebot commented Nov 26, 2020

analyze: analyze table after restore #605

analyze: analyze table after restore #605

Conversation

3pointer commented Nov 20, 2020

What problem does this PR solve?

What is changed and how it works?

Check List

Release Note

3pointer commented Nov 25, 2020

zhouqiang-cl commented Nov 25, 2020

3pointer commented Nov 25, 2020

3pointer commented Nov 25, 2020

kennytm Nov 25, 2020

Choose a reason for hiding this comment

3pointer Nov 25, 2020 • edited Loading

Choose a reason for hiding this comment

3pointer commented Nov 25, 2020

glorv commented Nov 25, 2020

glorv commented Nov 25, 2020

kennytm left a comment

Choose a reason for hiding this comment

3pointer commented Nov 25, 2020

glorv Nov 25, 2020

Choose a reason for hiding this comment

glorv Nov 25, 2020

Choose a reason for hiding this comment

glorv Nov 25, 2020

Choose a reason for hiding this comment

3pointer commented Nov 25, 2020

lichunzhu left a comment

Choose a reason for hiding this comment

3pointer commented Nov 26, 2020

glorv commented Nov 26, 2020

glorv commented Nov 26, 2020

kennytm left a comment

Choose a reason for hiding this comment

kennytm commented Nov 26, 2020

glorv commented Nov 26, 2020

ti-srebot commented Nov 26, 2020

ti-srebot commented Nov 26, 2020

3pointer Nov 25, 2020 •

edited

Loading