planner,executor,infoschema: add system table `table_storage_stats` #15056

reafans · 2020-03-02T02:46:20Z

What problem does this PR solve?

we can only get the disk-usage from http API

What is changed and how it works?

This pr could help us by using the table_storage_stats get the usage of a table, if a table is empty or its size < 0, its TABLE_SIZE would return 1MB.
Extract the get PdRegionStats func to helper which used by http API and this sys table.

mysql> select * from table_storage_stats where table_schema = 'test';
+--------------+------------+----------+------------+--------------+--------------------+------------+------------+
| TABLE_SCHEMA | TABLE_NAME | TABLE_ID | PEER_COUNT | REGION_COUNT | EMPTY_REGION_COUNT | TABLE_SIZE | TABLE_KEYS |
+--------------+------------+----------+------------+--------------+--------------------+------------+------------+
| test         | t          |      143 |          3 |            1 |                  1 |          1 |          0 |
| test         | usertable  |      343 |          3 |           97 |                  0 |       9127 |   16605517 |
| test         | t2         |      348 |          3 |            1 |                  1 |          1 |          0 |
| test         | tcs1       |      360 |          3 |           70 |                  0 |       7087 |   10204927 |
| test         | tcs2       |      362 |          3 |           70 |                  0 |       7141 |    9712408 |
| test         | tcs3       |      364 |          3 |           70 |                  0 |       7174 |    9765219 |
+--------------+------------+----------+------------+--------------+--------------------+------------+------------+
6 rows in set (0.35 sec)

mysql> select * from information_schema.table_storage_stats where table_schema = 'test' and table_name='t';
+--------------+------------+----------+------------+--------------+--------------------+------------+------------+
| TABLE_SCHEMA | TABLE_NAME | TABLE_ID | PEER_COUNT | REGION_COUNT | EMPTY_REGION_COUNT | TABLE_SIZE | TABLE_KEYS |
+--------------+------------+----------+------------+--------------+--------------------+------------+------------+
| test         | t          | 441      | 3          | 1            | 1                  | 1          | 0          |
+--------------+------------+----------+------------+--------------+--------------------+------------+------------+
1 row in set

Check List

Tests

Integration test

Release note

Add system table table_storage_stats

codecov · 2020-03-02T03:07:39Z

Codecov Report

Merging #15056 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #15056   +/-   ##
===========================================
  Coverage   79.6451%   79.6451%           
===========================================
  Files           520        520           
  Lines        141308     141308           
===========================================
  Hits         112545     112545           
  Misses        19773      19773           
  Partials       8990       8990

…nto add_disk_usage_tb

djshow832

No test cases?

executor/infoschema_reader.go

breezewish · 2020-03-09T04:20:36Z

Maybe we should integrate it with INFORMATION_SCHEMA.TABLES as much as possible, to be compatible with different kind of MySQL tools.

executor/infoschema_reader.go

reafans · 2020-03-10T13:21:13Z

/run-all-tests

reafans · 2020-03-10T13:27:31Z

/run-unit-test

zimulala · 2020-05-13T03:46:50Z

executor/infoschema_reader.go

+
+	// Extract the tables to the initialTable.
+	for _, db := range databases {
+		// The user didn't specified the table, extract all tables of this db to initialTable.


zimulala · 2020-05-13T03:47:02Z

executor/infoschema_reader.go

+		}
+	}
+
+	// Cache the helper and return an error if pd unavailable.


zimulala · 2020-05-13T03:50:09Z

executor/infoschema_reader.go

+	rows := make([][]types.Datum, 0, 1024)
+	count := 0
+	for e.curTable < len(e.initialTables) && count < 1024 {
+		table := (e.initialTables)[e.curTable]


Suggested change

table := (e.initialTables)[e.curTable]

table := e.initialTables[e.curTable]

zimulala

Rest LGTM

crazycs520

LGTM,
Please address comment.

…k_usage_tb

…nto add_disk_usage_tb

crazycs520 · 2020-05-13T06:37:37Z

/run-all-tests

crazycs520 · 2020-06-01T16:01:17Z

/run-all-tests

Signed-off-by: crazycs <crazycs520@gmail.com>

crazycs520 · 2020-06-02T06:15:36Z

/run-all-tests

crazycs520 · 2020-10-13T11:18:33Z

/run-cherry-picker

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2020-10-13T11:19:27Z

cherry pick to release-4.0 in PR #20431

…15056) (#20431) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Signed-off-by: crazycs520 <crazycs520@gmail.com>

reafans added 2 commits March 1, 2020 20:41

add disk_usage table

2e4ba13

fix bug

07a29b2

reafans requested a review from a team as a code owner March 2, 2020 02:46

ghost requested review from alivxxx and lzmhhh123 and removed request for a team March 2, 2020 02:46

reafans and others added 4 commits March 2, 2020 10:51

fix bug

6fe1583

fix lint

fc0d988

Merge branch 'master' into add_disk_usage_tb

9e3120a

fix bug

48a3ccd

Merge branch 'add_disk_usage_tb' of https://github.com/reafans/tidb i…

005f694

…nto add_disk_usage_tb

zz-jason changed the title ~~infoschema: add disk_usage sys table~~ planner,executor,infoschema: add system table disk_usage Mar 2, 2020

zz-jason added sig/execution SIG execution component/infoschema sig/planner SIG: Planner labels Mar 2, 2020

reafans added the status/WIP label Mar 2, 2020

djshow832 reviewed Mar 3, 2020

View reviewed changes

github-actions bot added the sig/sql-infra SIG: SQL Infra label Mar 8, 2020

breezewish reviewed Mar 9, 2020

View reviewed changes

executor/infoschema_reader.go Outdated Show resolved Hide resolved

reafans changed the title ~~planner,executor,infoschema: add system table disk_usage~~ planner,executor,infoschema: add system table table_storage_stats Mar 9, 2020

reafans added 2 commits March 10, 2020 21:15

update

4c084bb

update

34c7ce1

reafans requested a review from a team as a code owner March 10, 2020 13:16

ghost requested review from wshwsh12 and XuHuaiyu and removed request for a team March 10, 2020 13:16

zimulala reviewed May 13, 2020

View reviewed changes

crazycs520 reviewed May 13, 2020

View reviewed changes

reafans and others added 4 commits May 13, 2020 12:52

Merge branch 'master' of https://github.com/pingcap/tidb into add_dis…

a493eea

…k_usage_tb

Merge branch 'master' of https://github.com/pingcap/tidb into add_dis…

5caa82b

…k_usage_tb

Merge branch 'add_disk_usage_tb' of https://github.com/reafans/tidb i…

f8f07f9

…nto add_disk_usage_tb

Merge branch 'master' into add_disk_usage_tb

42cb402

crazycs520 and others added 3 commits May 13, 2020 14:56

Merge branch 'master' into add_disk_usage_tb

e45056a

Merge branch 'master' into add_disk_usage_tb

3ed9c2b

Merge branch 'master' into add_disk_usage_tb

a629ac1

crazycs520 previously approved these changes Jun 1, 2020

View reviewed changes

Merge branch 'master' into add_disk_usage_tb

d6f40c3

crazycs520 dismissed their stale review via d6f40c3 June 1, 2020 16:00

fix ci

1e1fdeb

Signed-off-by: crazycs <crazycs520@gmail.com>

crazycs520 approved these changes Jun 2, 2020

View reviewed changes

Merge branch 'master' into add_disk_usage_tb

ddf017a

crazycs520 added 2 commits June 2, 2020 15:30

Merge branch 'master' into add_disk_usage_tb

bcc8180

Merge branch 'master' into add_disk_usage_tb

c6b1412

crazycs520 merged commit cfbd754 into pingcap:master Jun 2, 2020

reafans deleted the add_disk_usage_tb branch June 3, 2020 02:25

crazycs520 added the needs-cherry-pick-4.0 label Oct 13, 2020

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Oct 13, 2020

cherry pick pingcap#15056 to release-4.0

adebfe7

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Oct 13, 2020

planner,executor,infoschema: add system table table_storage_stats (#15056) #20431

Merged

ti-srebot added a commit that referenced this pull request Oct 15, 2020

planner,executor,infoschema: add system table table_storage_stats (#…

06cbfe6

…15056) (#20431) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Signed-off-by: crazycs520 <crazycs520@gmail.com>

tangenta mentioned this pull request Jul 22, 2022

executor: refine the error message for TABLE_STORAGE_STATS #36464

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner,executor,infoschema: add system table `table_storage_stats` #15056

planner,executor,infoschema: add system table `table_storage_stats` #15056

reafans commented Mar 2, 2020 •

edited by crazycs520

Loading

codecov bot commented Mar 2, 2020 •

edited

Loading

djshow832 left a comment

breezewish commented Mar 9, 2020

reafans commented Mar 10, 2020

reafans commented Mar 10, 2020

zimulala May 13, 2020

zimulala May 13, 2020

zimulala May 13, 2020

zimulala left a comment

crazycs520 left a comment

crazycs520 commented May 13, 2020

crazycs520 commented Jun 1, 2020

crazycs520 commented Jun 2, 2020

crazycs520 commented Oct 13, 2020

ti-srebot commented Oct 13, 2020

	table := (e.initialTables)[e.curTable]
	table := e.initialTables[e.curTable]

planner,executor,infoschema: add system table table_storage_stats #15056

planner,executor,infoschema: add system table table_storage_stats #15056

Conversation

reafans commented Mar 2, 2020 • edited by crazycs520 Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

codecov bot commented Mar 2, 2020 • edited Loading

Codecov Report

djshow832 left a comment

Choose a reason for hiding this comment

breezewish commented Mar 9, 2020

reafans commented Mar 10, 2020

reafans commented Mar 10, 2020

zimulala May 13, 2020

Choose a reason for hiding this comment

zimulala May 13, 2020

Choose a reason for hiding this comment

zimulala May 13, 2020

Choose a reason for hiding this comment

zimulala left a comment

Choose a reason for hiding this comment

crazycs520 left a comment

Choose a reason for hiding this comment

crazycs520 commented May 13, 2020

crazycs520 commented Jun 1, 2020

crazycs520 commented Jun 2, 2020

crazycs520 commented Oct 13, 2020

ti-srebot commented Oct 13, 2020

planner,executor,infoschema: add system table `table_storage_stats` #15056

planner,executor,infoschema: add system table `table_storage_stats` #15056

reafans commented Mar 2, 2020 •

edited by crazycs520

Loading

codecov bot commented Mar 2, 2020 •

edited

Loading