UCP: Speed up slow_query

[crazycs520]

Description

TiDB has an SLOW_QUERY system table, it's data is parse from slow-log when user query this table.

Currently, TiDB user synchronous parsing slow-log file when parrent executor calling Next method. But we can
use another goroutine to Asynchronous parsing slow-log to speed up query SLOW_QUERY. the progress maybe like below:

func (e *slowQueryRetriever) initialize(sctx sessionctx.Context) error {
	...
	go e.parseSlowLog()
	...
}

func (e *slowQueryRetriever) retrieve(ctx context.Context, sctx sessionctx.Context) ([][]types.Datum, error) {
	...
	// get rows from cache channel
	rows := <- e.resultCh
	...
}

Score

• 200

Mentor(s)

• @crazycs

Contact the mentors: #tidb-challenge-program channel in TiDB Community Slack Workspace

Recommended Skills

• go

Learning Materials

• 慢查询文档

[ziyi-yan]

/pick-up-challenge

[sre-bot]

@ziyi-yan pick up issue success

[ziyi-yan]

Hi, @crazycs520. I found out that the retrieve() method is only called by memTableRetriever interface at here (other than test code). If we want to achieve this optimization, we might need to call initialize() when we construct the slowQueryRetriever, such as here:

tidb/executor/builder.go

Lines 1428 to 1435 in 9e98f70

	case strings.ToLower(infoschema.TableSlowQuery), strings.ToLower(infoschema.ClusterTableSlowLog):
	return &MemTableReaderExec{
	baseExecutor: newBaseExecutor(b.ctx, v.Schema(), v.ExplainID()),
	retriever: &slowQueryRetriever{
	table: v.Table,
	outputCols: v.Columns,
	extractor: v.Extractor.(*plannercore.SlowQueryExtractor),
	},

Is it OK to put some initialization code here?

[ziyi-yan]

I just submit a draft PR for further discussion. It seems that it's kind of hard to do initialization when constructing the slowQueryRetriever, because we cannot return error in buildMemTable(): https://github.com/pingcap/tidb/pull/15363/files#diff-badfcd30d7596a08cd207b8e6ae778e6R1448-R1453