Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

planner: extract a LogicalMemTable from DataSource to decouple memory/stored tables #13741

Merged
merged 3 commits into from
Nov 27, 2019
Merged

planner: extract a LogicalMemTable from DataSource to decouple memory/stored tables #13741

merged 3 commits into from
Nov 27, 2019

Conversation

lonng
Copy link
Contributor

@lonng lonng commented Nov 26, 2019

Signed-off-by: Lonng heng@lonng.org

What problem does this PR solve?

In the current implementation, all tables will be represented as a DataSource in the logical plan build stage. But tables can be classified into the virtual table (memory table which data are stored in memory) and the stored table (data are stored in TiKV). And in our #13567 feature need to push some predictions down to the virtual table to make data retrieval more efficiency.

e.g:

SELECT * FROM tidb_cluster_log WHERE type='tikv' AND address='192.16.5.32'

In the current implementation, we must retrieve all logs from cluster components and filter them in TiDB by predications type='tikv' AND address='192.16.5.32', which is inefficiency. And the more efficient way is that push type='tikv' AND address='192.16.5.32' to the lowest virtual table tidb_cluster_log and make the LogReaderExec have some information about predications.

This PR split DataSource into DataSource and LogicalMemTable and make the LogicalMemTable independent to make the subsequent work easily and don't use any tricky/hack ways to achieve our goal.

What is changed and how it works?

This PR extracts a LogicalMemTable from DataSource to decouple memory/stored tables and make some predications push-down to memory table more easily.

Check List

Tests

  • Exists Unit test

Release note

  • No need

@lonng lonng added the sig/planner SIG: Planner label Nov 26, 2019
@codecov
Copy link

codecov bot commented Nov 26, 2019

Codecov Report

Merging #13741 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #13741   +/-   ##
===========================================
  Coverage   80.2815%   80.2815%           
===========================================
  Files           474        474           
  Lines        117631     117631           
===========================================
  Hits          94436      94436           
  Misses        15840      15840           
  Partials       7355       7355

Copy link
Member

@winoros winoros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Copy link
Contributor

@crazycs520 crazycs520 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@crazycs520
Copy link
Contributor

/run-all-tests

@crazycs520
Copy link
Contributor

/run-all-tests

@lonng lonng added the status/LGT2 Indicates that a PR has LGTM 2. label Nov 27, 2019
@lonng lonng merged commit 6b7e19a into pingcap:master Nov 27, 2019
@lonng lonng deleted the logical-mem-table branch November 27, 2019 02:19
qw4990 pushed a commit to qw4990/tidb that referenced this pull request Nov 27, 2019
XiaTianliang pushed a commit to XiaTianliang/tidb that referenced this pull request Dec 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/planner SIG: Planner status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants