infoschema,planner: implement the metric table predicate push down #14169

crazycs520 · 2019-12-20T14:51:14Z

What problem does this PR solve?

Implement the metric table predicate push down.

mysql>desc select * from up;
+----------------+----------+------+--------------------------------------------------------------------------------------------+
| id             | count    | task | operator info                                                                              |
+----------------+----------+------+--------------------------------------------------------------------------------------------+
| MemTableScan_4 | 10000.00 | root | PromQL:up{}, start_time:2019-12-20 22:26:13.47, end_time:2019-12-20 22:26:13.47, step:1m0s |
+----------------+----------+------+--------------------------------------------------------------------------------------------+
1 row in set
Time: 0.006s
mysql>desc select * from up where time > "2019-12-20 22:10:13"
+----------------+----------+------+------------------------------------------------------------------------------------------+
| id             | count    | task | operator info                                                                            |
+----------------+----------+------+------------------------------------------------------------------------------------------+
| MemTableScan_5 | 10000.00 | root | PromQL:up{}, start_time:2019-12-20 22:10:13, end_time:2019-12-20 22:26:28.558, step:1m0s |
+----------------+----------+------+------------------------------------------------------------------------------------------+
1 row in set
Time: 0.006s
mysql>desc select * from query_duration where `Quantile` = 0.8 and time > "2019-12-20 22:10:13"
+----------------+----------+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id             | count    | task | operator info                                                                                                                                                                           |
+----------------+----------+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| MemTableScan_5 | 10000.00 | root | PromQL:histogram_quantile(0.8, sum(rate(tidb_server_handle_query_duration_seconds_bucket{}[60s])) by (le)), start_time:2019-12-20 22:10:13, end_time:2019-12-20 22:26:39.944, step:1m0s |
+----------------+----------+------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set
Time: 0.009s
mysql>desc select * from query_duration where `Quantile` = 0.8 and instance="127.0.0.1:10080" and sql_type="general";
+----------------+----------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id             | count    | task | operator info                                                                                                                                                                                                                            |
+----------------+----------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| MemTableScan_5 | 10000.00 | root | PromQL:histogram_quantile(0.8, sum(rate(tidb_server_handle_query_duration_seconds_bucket{instance="127.0.0.1:10080",sql_type="general"}[60s])) by (le)), start_time:2019-12-20 22:27:06.407, end_time:2019-12-20 22:27:06.407, step:1m0s |
+----------------+----------+------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
mysql>select * from query_duration where `Quantile` = 0.8 and sql_type="internal" and time > "2019-12-20 22:10:13";
+---------------------+-------------------+----------+----------+----------+
| time                | value             | instance | sql_type | Quantile |
+---------------------+-------------------+----------+----------+----------+
| 2019-12-20 22:10:13 | 0.00126666666667  |          | internal | 0.8      |
| 2019-12-20 22:11:13 | 0.00136           |          | internal | 0.8      |
| 2019-12-20 22:13:13 | 0.001             |          | internal | 0.8      |
| 2019-12-20 22:14:13 | 0.00133333333333  |          | internal | 0.8      |
| 2019-12-20 22:15:13 | 0.0009            |          | internal | 0.8      |
| 2019-12-20 22:16:13 | 0.00133333333333  |          | internal | 0.8      |
| 2019-12-20 22:17:13 | 0.000961538461538 |          | internal | 0.8      |
| 2019-12-20 22:18:13 | 0.000957142857143 |          | internal | 0.8      |
| 2019-12-20 22:19:13 | 0.00166666666667  |          | internal | 0.8      |
| 2019-12-20 22:20:13 | 0.000928571428571 |          | internal | 0.8      |
| 2019-12-20 22:21:13 | 0.000928571428571 |          | internal | 0.8      |
| 2019-12-20 22:22:13 | 0.0016            |          | internal | 0.8      |
| 2019-12-20 22:27:13 | 0.0009            |          | internal | 0.8      |
| 2019-12-20 22:28:13 | 0.0014            |          | internal | 0.8      |
| 2019-12-20 22:29:13 | 0.000992307692308 |          | internal | 0.8      |
| 2019-12-20 22:30:13 | 0.000961538461538 |          | internal | 0.8      |
| 2019-12-20 22:31:13 | 0.000928571428571 |          | internal | 0.8      |
| 2019-12-20 22:32:13 | 0.000928571428571 |          | internal | 0.8      |
| 2019-12-20 22:33:13 | 0.0009            |          | internal | 0.8      |
| 2019-12-20 22:34:13 | 0.001             |          | internal | 0.8      |
| 2019-12-20 22:35:13 | 0.002             |          | internal | 0.8      |
| 2019-12-20 22:36:13 | 0.0009            |          | internal | 0.8      |
| 2019-12-20 22:37:13 | 0.000928571428571 |          | internal | 0.8      |
| 2019-12-20 22:38:13 | 0.0015            |          | internal | 0.8      |
| 2019-12-20 22:39:13 | 0.000992307692308 |          | internal | 0.8      |
| 2019-12-20 22:40:13 | 0.0015            |          | internal | 0.8      |
| 2019-12-20 22:41:13 | 0.001             |          | internal | 0.8      |
| 2019-12-20 22:42:13 | 0.000928571428571 |          | internal | 0.8      |
| 2019-12-20 22:43:13 | 0.00226666666667  |          | internal | 0.8      |
| 2019-12-20 22:44:13 | 0.001             |          | internal | 0.8      |
| 2019-12-20 22:45:13 | 0.000928571428571 |          | internal | 0.8      |
| 2019-12-20 22:46:13 | 0.0009            |          | internal | 0.8      |
+---------------------+-------------------+----------+----------+----------+
32 rows in set

What is changed and how it works?

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

Code changes

Has exported function/method change

…table-extractor

planner/core/memtable_predicate_extractor.go

planner/core/explain.go

djshow832 · 2019-12-23T06:59:16Z

planner/core/memtable_predicate_extractor.go

+
+func (e *MetricTableExtractor) convertToTime(t int64) time.Time {
+	if t == 0 || t == math.MaxInt64 {
+		return time.Now()


What if startTime=0, endTime=now-1h?
You changed it to [now, now-1h].

Yes, but it will got error when query.

mysql>desc select * from query_duration where `Quantile` in ( 0.99) and sql_type="internal" and time < "2019-12-23 16:10:13" and instance="127.0.0.1:10080"; +----------------+----------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | id | count | task | operator info | +----------------+----------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | MemTableScan_5 | 10000.00 | root | PromQL:histogram_quantile(0.99, sum(rate(tidb_server_handle_query_duration_seconds_bucket{instance="127.0.0.1:10080",sql_type="internal"}[60s])) by (le)), start_time:2019-12-23 16:20:47.559, end_time:2019-12-23 16:10:12, step:1m0s | +----------------+----------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set Time: 0.009s mysql>select * from query_duration where `Quantile` in ( 0.99) and sql_type="internal" and time < "2019-12-23 16:10:13" and instance="127.0.0.1:10080"; (1105, u'bad_data: end timestamp must not be before start time')

I mean, it shouldn't return an error.
When user queries [0, 16:10], you can change it to [13:40, 16:10], not [16:20, 16:10].

addressed. But When the user queries [0, 16:10], it will change it to [16:10, 16:10] now.

You can refer to
https://github.com/pingcap/tidb/blob/master/executor/cluster_reader.go#L469

infoschema/metricschema/table.go

planner/core/memtable_predicate_extractor.go

executor/metric_reader.go

infoschema/metricschema/tables.go

executor/metric_reader.go

executor/cluster_reader_test.go

executor/metric_reader.go

…0/tidb into metric-table-extractor

executor/metric_reader.go

infoschema/metricschema/table.go

planner/core/memtable_predicate_extractor.go

infoschema/metricschema/table.go

executor/metric_reader.go

planner/core/explain.go

planner/core/memtable_predicate_extractor.go

planner/core/explain.go

planner/core/memtable_predicate_extractor.go

lonng

Rest LGTM

crazycs520 · 2019-12-25T02:21:40Z

/run-all-tests

lonng

LGTM

crazycs520 · 2019-12-25T02:42:22Z

/run-unit-test

djshow832

LGTM

crazycs520 added 5 commits December 19, 2019 20:45

init

1f39533

Merge branch 'master' of https://github.com/pingcap/tidb into metric-…

7c1777c

…table-extractor

pushdown selection

f1029c6

add test

ed2c8b4

refine test

1a68c68

crazycs520 added the type/usability label Dec 20, 2019

crazycs520 requested a review from a team as a code owner December 20, 2019 14:51

ghost requested review from alivxxx and lzmhhh123 and removed request for a team December 20, 2019 14:51

lonng reviewed Dec 21, 2019

View reviewed changes

planner/core/memtable_predicate_extractor.go Show resolved Hide resolved

lonng reviewed Dec 21, 2019

View reviewed changes

planner/core/memtable_predicate_extractor.go Outdated Show resolved Hide resolved

djshow832 reviewed Dec 23, 2019

View reviewed changes

address comment

193fd30

lonng reviewed Dec 23, 2019

View reviewed changes

infoschema/metricschema/table.go Outdated Show resolved Hide resolved

lonng reviewed Dec 23, 2019

View reviewed changes

planner/core/memtable_predicate_extractor.go Outdated Show resolved Hide resolved

lonng reviewed Dec 23, 2019

View reviewed changes

planner/core/memtable_predicate_extractor.go Outdated Show resolved Hide resolved

lonng reviewed Dec 23, 2019

View reviewed changes

executor/metric_reader.go Outdated Show resolved Hide resolved

lonng reviewed Dec 23, 2019

View reviewed changes

infoschema/metricschema/tables.go Show resolved Hide resolved

address comment

f3cd4e9

lonng reviewed Dec 23, 2019

View reviewed changes

executor/metric_reader.go Show resolved Hide resolved

crazycs520 added 2 commits December 23, 2019 17:48

add failpoint test

4116d7c

fix ci

1c36083

lonng reviewed Dec 23, 2019

View reviewed changes

executor/cluster_reader_test.go Outdated Show resolved Hide resolved

lonng reviewed Dec 23, 2019

View reviewed changes

executor/metric_reader.go Outdated Show resolved Hide resolved

address comment

56ecbc6

crazycs520 force-pushed the metric-table-extractor branch from fd13db4 to 56ecbc6 Compare December 23, 2019 11:47

crazycs520 added 3 commits December 23, 2019 20:08

Merge branch 'master' into metric-table-extractor

6a7082f

make test stable

f6a8afc

Merge branch 'metric-table-extractor' of https://github.com/crazycs52…

ae4aede

…0/tidb into metric-table-extractor

lonng reviewed Dec 23, 2019

View reviewed changes

executor/metric_reader.go Outdated Show resolved Hide resolved

lonng reviewed Dec 23, 2019

View reviewed changes

infoschema/metricschema/table.go Outdated Show resolved Hide resolved

lonng reviewed Dec 23, 2019

View reviewed changes

planner/core/memtable_predicate_extractor.go Outdated Show resolved Hide resolved

fix timezone

dc0d4e9

lonng reviewed Dec 23, 2019

View reviewed changes

infoschema/metricschema/table.go Outdated Show resolved Hide resolved

crazycs520 added 3 commits December 23, 2019 21:31

address comment

1e0721d

address comment

2c91a35

address comment

1465a6e

lonng reviewed Dec 24, 2019

View reviewed changes

executor/metric_reader.go Outdated Show resolved Hide resolved

lonng reviewed Dec 24, 2019

View reviewed changes

planner/core/explain.go Outdated Show resolved Hide resolved

lonng reviewed Dec 24, 2019

View reviewed changes

planner/core/explain.go Outdated Show resolved Hide resolved

lonng reviewed Dec 24, 2019

View reviewed changes

planner/core/explain.go Outdated Show resolved Hide resolved

lonng reviewed Dec 24, 2019

View reviewed changes

planner/core/memtable_predicate_extractor.go Outdated Show resolved Hide resolved

lonng reviewed Dec 24, 2019

View reviewed changes

planner/core/memtable_predicate_extractor.go Outdated Show resolved Hide resolved

crazycs520 added 3 commits December 24, 2019 15:04

address comment

280014a

Merge branch 'master' into metric-table-extractor

b115768

address comment of default time range

27e7951

lonng reviewed Dec 24, 2019

View reviewed changes

planner/core/explain.go Outdated Show resolved Hide resolved

lonng reviewed Dec 24, 2019

View reviewed changes

planner/core/memtable_predicate_extractor.go Outdated Show resolved Hide resolved

lonng reviewed Dec 24, 2019

View reviewed changes

crazycs520 added 2 commits December 25, 2019 10:20

address comment

d0c5df3

Merge branch 'master' into metric-table-extractor

7a0a782

lonng reviewed Dec 25, 2019

View reviewed changes

djshow832 reviewed Dec 25, 2019

View reviewed changes

djshow832 approved these changes Dec 25, 2019

View reviewed changes

djshow832 merged commit 279ad0d into pingcap:master Dec 25, 2019

lonng mentioned this pull request Dec 25, 2019

Tracking issue for TiDB built-in SQL Diagnostics #13567

Closed

56 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infoschema,planner: implement the metric table predicate push down #14169

infoschema,planner: implement the metric table predicate push down #14169

crazycs520 commented Dec 20, 2019

djshow832 Dec 23, 2019

crazycs520 Dec 23, 2019

djshow832 Dec 23, 2019

crazycs520 Dec 23, 2019

djshow832 Dec 24, 2019

crazycs520 Dec 24, 2019

lonng left a comment

crazycs520 commented Dec 25, 2019

lonng left a comment

crazycs520 commented Dec 25, 2019

djshow832 left a comment

infoschema,planner: implement the metric table predicate push down #14169

infoschema,planner: implement the metric table predicate push down #14169

Conversation

crazycs520 commented Dec 20, 2019

What problem does this PR solve?

What is changed and how it works?

Check List

djshow832 Dec 23, 2019

Choose a reason for hiding this comment

crazycs520 Dec 23, 2019

Choose a reason for hiding this comment

djshow832 Dec 23, 2019

Choose a reason for hiding this comment

crazycs520 Dec 23, 2019

Choose a reason for hiding this comment

djshow832 Dec 24, 2019

Choose a reason for hiding this comment

crazycs520 Dec 24, 2019

Choose a reason for hiding this comment

lonng left a comment

Choose a reason for hiding this comment

crazycs520 commented Dec 25, 2019

lonng left a comment

Choose a reason for hiding this comment

crazycs520 commented Dec 25, 2019

djshow832 left a comment

Choose a reason for hiding this comment