*: re-implement partition pruning for better performance

[tiancaiamao]

What problem does this PR solve?

Re-implement partition pruning for better performance.

The old partition pruning's implementation uses an algorithm we called constraint propagate which is powerful, yet slow.

It works like this:

For each partition:
    construct EXPR =  (partition's expression AND query's expression)
    if FixPoint(constraint propagate EXPR) == AlwaysFalse
            Prune

It's powerful as it can propagate something like a > b and b > 3 => a > 3 and it also include some propagate rules for functions x > const1, f(x) < const2, f is monotonous => false.

It's possible to handle some cases like:

create table t (a int, b int) partition by (a) ...
select * from t where a > b and b > 3;

But the process is slow because constructing new expressions involve too many object allocation, and the constraint propagation process is also heavy. If there are many partitions, 2048 for example, the whole time spent on partition pruning would be significant.

What is changed and how it works?

The new partition pruning algorithm is much faster by using binary search and avoid constructing expressions.

The query expression would be abstract to f(col) op const, where op is one of > < = >= <=, in the code it's represent as dataForPrune.

The 'partition p0 less than xxx, partition p1 less than xxx, ...' forms an array [p0 p1 p2 ... maxvalue], represented as the lessThanData.

The new algorithm uses a binary search pruneUseBinarySearch to locate const in the array.

A simple benchmark,
Before:

session git:(partition-prune) ✗ go test -test.bench BenchmarkPartitionPruning -test.run Ignore
BenchmarkPartitionPruning-4   	       8	 128010294 ns/op

After:

session git:(partition-prune) ✗ go test -test.bench BenchmarkPartitionPruning -test.run Ignore
BenchmarkPartitionPruning-4   	      21	  57979203 ns/op

---

Some side notes:

1. As null values are all located in the first partition, the first partition expression in the old implementation is val < xxx or val is null. The or val is null condition is hard to eliminate so we can't prune the first partition in many cases.
   The new implementation doesn't consider the null value, so the first partition is pruned.

2. There is a relax operation during the pruning, and it may lead to less accurate pruning results.
   For example,

create table t (a datetime) partition by to_days(a) ...

This condition doesn't always hold:

a < const => to_days(a) < to_days(const)

A counterexample is:

2020-02-12 10:08:00 < 2020-02-12 23:59:59 => to_days(2020-02-12 10:08:00) = to_days(2020-02-12 23:59:59)

So we have to relax < to <= to handle functions.

a < const => to_days(a) <= to_days(const)

Check List

Tests

• Unit test

Related changes

• Need to cherry-pick to the release branch

Release note

• Write release note for bug-fix or new feature.

[tiancaiamao]

/run-unit-test

PTAL @imtbkcat @zz-jason

[imtbkcat]

LGTM

[tiancaiamao]

Just now I run a benchmark on my own laptop and compare the result
(latency in milliseconds, the lower the better).

The old version:

Mean:  3650.8462739778993
Quantile 80:  4975.331340000001
Quantile 95:  7505.103714999999
Quantile 99:  9457.780304
Total Succ: 181

and the new version:

Mean:  2072.399426448846
Quantile 80:  3150.99298025
Quantile 95:  4735.244528499999
Quantile 99:  6287.017979
Total Succ: 303

[lysu]

rest LGTM

[lysu]

Suggested change

	return partitionRangeOR(reduceAllocation[:1])
	return reduceAllocation[:1]

[lysu]

just a question: can we make sure or be sorted?

if so maybe we can more optimize to avoid {6, 7} U {3, 9} call in {0, 4}, {6, 7}, {8, 11} U {3, 9} or {3, +INF} U {6, 7} call in {3, +INF} U {0, 4}, {6, 7}- -?

it seems leaf exp (e.g. a > 3 in a > 3 and b < 5) can take benfit from this(although intersectionRange is fast)

[tiancaiamao]

Maintain the sorted condition is more restrictive.
The or array is usually small, so I guess loop over the whole array VS maintain the sorted array and break in advance don't have a big difference.

[lysu]

LGTM