stats: fix histogram boundaries overflow error #7883

alivxxx · 2018-10-12T12:16:51Z

What problem does this PR solve?

When loading histogram from the store, it will raise an error like constant -9223372036854775808 overflows int.
Fix #7868

What is changed and how it works?

The out of range value comes from the feedback. We should adjust their values according to the column type info.

Check List

Tests

Unit test

Code changes

Has exported function/method change

Side effects

If the error already happens, we need to first analyze the table.

Related changes

Need to cherry-pick to the release branch

PTAL @zz-jason @winoros @eurekaka

eurekaka

Why we would generate feedback outside the range of column type and write it to store, shall we prevent this out of range from the root?

eurekaka · 2018-10-12T12:55:06Z

statistics/feedback.go

 	total := 0
-	for _, ran := range feedback.feedback {
-		idx, _ := h.Bounds.LowerBound(0, ran.lower)
+	sc := &stmtctx.StatementContext{TimeZone: time.UTC}


The TimeZone is for functions like compareMysqlTime? the hard code looks confusing...

Because the timeZone does not matter here, so we just do not leave it as nil.

eurekaka · 2018-10-12T13:13:45Z

statistics/feedback.go

@@ -528,7 +587,12 @@ func splitBuckets(h *Histogram, feedback *QueryFeedback) ([]bucket, []bool, int6
 func UpdateHistogram(h *Histogram, feedback *QueryFeedback) *Histogram {
 	buckets, isNewBuckets, totalCount := splitBuckets(h, feedback)
 	buckets = mergeBuckets(buckets, isNewBuckets, float64(totalCount))
-	return buildNewHistogram(h, buckets)
+	hist := buildNewHistogram(h, buckets)
+	// Update the NDV of primary key column.


Why do we move NDV update to here?

The origin place is not correct, it is before the update of the histogram.

Here is more natural, if we update the histogram, we'd better update all the info here.

alivxxx · 2018-10-15T03:11:17Z

@eurekaka Yes, but maybe that is another problem: for example, when we have a primary key of the handle, the query range of a < 1 will be transformed to (minInt64, 1), no matter the type of primary key, so feedback will get the lower bound as minInt64.

eurekaka

LGTM

zz-jason

LGTM

alivxxx · 2018-10-17T05:50:18Z

/run-all-tests

bugfix fixed pingcap#7518 expression: MySQL compatible current_user function (pingcap#7801) plan: propagate constant over outer join (pingcap#7794) - extract `outerCol = const` from join conditions and filter conditions, substitute `outerCol` in join conditions with `const`; - extract `outerCol = innerCol` from join conditions, derive new join conditions based on this column equal condition and `outerCol` related expressions in join conditions and filter conditions; util/timeutil: fix data race caused by forgetting set stats lease to 0 (pingcap#7901) stats: handle ddl event for partition table (pingcap#7903) plan: implement Operand and Pattern of cascades planner. (pingcap#7910) planner: not convert to TableDual if empty range is derived from deferred constants (pingcap#7808) plan: move projEliminate behind aggEliminate (pingcap#7909) admin: fix admin check table bug of byte compare (pingcap#7887) * admin: remove reflect deepEqual stats: fix panic caused by empty histogram (pingcap#7912) plan: fix panic caused by empty schema of LogicalTableDual (pingcap#7906) * fix drop view if exist error (pingcap#7833) executor: refine `explain analyze` (pingcap#7888) executor: add an variable to compatible with MySQL insert for OGG (pingcap#7863) expression: maintain `DeferredExpr` in aggressive constant folding. (pingcap#7915) stats: fix histogram boundaries overflow error (pingcap#7883) ddl:support the definition of `null` change to `not null` using `alter table` (pingcap#7771) * ddl:support the definition of null change to not null using alter table ddl: add check when create table with foreign key. (pingcap#7885) * ddl: add check when create table with foreign key planner: eliminate if null on non null column (pingcap#7924) executor: fix a bug in point get (pingcap#7934) planner, executor: refine ColumnPrune for LogicalUnionAll (pingcap#7930) executor: fix panic when limit is too large (pingcap#7936) ddl: add TiDB version to metrics (pingcap#7902) stats: limit the length of sample values (pingcap#7931) vendor: update tipb (pingcap#7893) planner: support the Group and GroupExpr for the cascades planner (pingcap#7917) store/tikv: log more information when other err occurs (pingcap#7948) types: fix date time parse (pingcap#7933) ddl: just print error message when ddl job is normal to calcel, to eliminate noisy log (pingcap#7875) stats: update delta info for partition table (pingcap#7947) explaintest: add explain test for partition pruning (pingcap#7505) util: move disjoint set to util package (pingcap#7950) util: add PreAlloc4Row and Insert for Chunk and List (pingcap#7916) executor: add the slow log for commit (pingcap#7951) expression: add builtin json_keys (pingcap#7776) privilege: add USAGE in `show grants` for mysql compatibility (pingcap#7955) ddl: fix invailid ddl job panic (pingcap#7940) *: move ast.NewValueExpr to standalone parser_driver package (pingcap#7952) Make the ast package get rid of the dependency of types.Datum server: allow cors http request (pingcap#7939) *: move `Statement` and `RecordSet` from ast to sqlexec package (pingcap#7970) pr suggestion update executor/aggfuncs: split unit tests to corresponding file (pingcap#7993) store/tikv: fix typo (pingcap#7990) executor, planner: clone proj schema for different children in buildProj4Union (pingcap#7999) executor: let information_schema be the first database in ShowDatabases (pingcap#7938) stats: use local feedback for partition table (pingcap#7963) executor: add unit test for aggfuncs (pingcap#7966) server: add log for binary execute statement (pingcap#7987) admin: refine admin check decoder (pingcap#7862) executor: improve wide table insert & update performance (pingcap#7935) ddl: fix reassigned partition id in `truncate table` does not take effect (pingcap#7919) fix reassigned partition id in truncate table does not take effect add changelog for 2.1.0 rc4 (pingcap#8020) *: make parser package dependency as small as possible (pingcap#7989) parser: support `:=` in the `set` syntax (pingcap#8018) According to MySQL document, `set` use the = assignment operator, but the := assignment operator is also permitted stats: garbage collect stats for partition table (pingcap#7962) docs: add the proposal for the column pool (pingcap#7988) expression: refine built-in func truncate to support uint arg (pingcap#8000) stats: support show stats for partition table (pingcap#8023) stats: update error rate for partition table (pingcap#8022) stats: fix estimation for out of range point queries (pingcap#8015) *: move parser to a separate repository (pingcap#8036) executor: fix wrong result when index join on union scan. (pingcap#8031) Do not modify Plan of dataReaderBuilder directly, because it would impact next batch of outer rows, as well as other concurrent inner workers. Instead, build a local child builder to store the child plan. planner: fix a panic of a cached prepared statement with IndexScan (pingcap#8017) *: fix the issue of executing DDL after executing SQL failure in txn (pingcap#8044) * ddl, executor: fix the issue of executing DDL after executing SQL failure in txn add unit test remove debug info add like evaluator case sensitive test ddl, domain: make schema correct after canceling jobs (pingcap#7997) unit test fix code format proposal: maintaining histograms in plan. (pingcap#7605) support _tidb_rowid for table scan range (pingcap#8047) var rename fix

stats: fix feedback out of range error

4cc2454

alivxxx added component/statistics type/bugfix This PR fixes a bug. labels Oct 12, 2018

eurekaka reviewed Oct 12, 2018

View reviewed changes

Merge branch 'master' into range

89ee4b2

eurekaka reviewed Oct 15, 2018

View reviewed changes

eurekaka added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 15, 2018

eurekaka and others added 2 commits October 15, 2018 13:33

Merge branch 'master' into range

647b2c5

Merge branch 'master' into range

795b25c

zz-jason approved these changes Oct 17, 2018

View reviewed changes

zz-jason added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 17, 2018

Merge branch 'master' into range

2283f87

alivxxx added the status/all tests passed label Oct 17, 2018

alivxxx merged commit a7f78c7 into pingcap:master Oct 17, 2018

alivxxx deleted the range branch October 17, 2018 06:45

alivxxx added a commit to alivxxx/tidb that referenced this pull request Oct 18, 2018

stats: fix histogram boundaries overflow error (pingcap#7883)

ffb71d1

alivxxx mentioned this pull request Oct 18, 2018

stats: fix histogram boundaries overflow error (#7883) #7944

Merged

zz-jason pushed a commit that referenced this pull request Oct 18, 2018

stats: fix histogram boundaries overflow error (#7883) (#7944)

4021862

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stats: fix histogram boundaries overflow error #7883

stats: fix histogram boundaries overflow error #7883

alivxxx commented Oct 12, 2018 •

edited

Loading

eurekaka left a comment

eurekaka Oct 12, 2018

alivxxx Oct 15, 2018

eurekaka Oct 12, 2018

alivxxx Oct 15, 2018

alivxxx commented Oct 15, 2018

eurekaka left a comment

zz-jason left a comment

alivxxx commented Oct 17, 2018

stats: fix histogram boundaries overflow error #7883

stats: fix histogram boundaries overflow error #7883

Conversation

alivxxx commented Oct 12, 2018 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

eurekaka left a comment

Choose a reason for hiding this comment

eurekaka Oct 12, 2018

Choose a reason for hiding this comment

alivxxx Oct 15, 2018

Choose a reason for hiding this comment

eurekaka Oct 12, 2018

Choose a reason for hiding this comment

alivxxx Oct 15, 2018

Choose a reason for hiding this comment

alivxxx commented Oct 15, 2018

eurekaka left a comment

Choose a reason for hiding this comment

zz-jason left a comment

Choose a reason for hiding this comment

alivxxx commented Oct 17, 2018

alivxxx commented Oct 12, 2018 •

edited

Loading