util: increase column's nullBitmap's capacity #12470

H-ZeX · 2019-09-29T05:15:48Z

Signed-off-by: H-ZeX hzx20112012@gmail.com

What problem does this PR solve?

`nullBitmap`'s cap

The origin nullBitmap's capacity is a little small.
When the cap param is (x<<3)+y(0<y<8), the nullBitMap's len is only x, so the col that it can represent is (x<<3), which is small that the column number (x<<3)+y, then this slice need to be expanded.
For example, if there is 8+7=15 col, we need 2 bytes nullBitmap to represent it. But, 15>>3 only get 1, so this slice need to be expanded to 2 bytes.
If the slice expands, the memory usage increased is not little, so I think adding one byte to nullBitmap is better and necessary.
In util/chunk/column.go, there is many patterns like (x+7)>>3 such as

func (c *Column) appendMultiSameNullBitmap(notNull bool, num int) {
	numNewBytes := ((c.length + num + 7) >> 3) - len(c.nullBitmap)

sizeNulls := (n + 7) >> 3

What is changed and how it works?

x>>3 -> ((x+7)>>3)

Check List

Tests

existing unit test

Side effects

No.

Related changes

No.

Release note

TODO

Signed-off-by: H-ZeX <hzx20112012@gmail.com>

winkyao · 2019-09-29T07:10:51Z

/rebuild

Signed-off-by: H-ZeX <hzx20112012@gmail.com>

codecov · 2019-09-29T08:17:08Z

Codecov Report

Merging #12470 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #12470   +/-   ##
===========================================
  Coverage   80.0167%   80.0167%           
===========================================
  Files           465        465           
  Lines        107330     107330           
===========================================
  Hits          85882      85882           
  Misses        14955      14955           
  Partials       6493       6493

winkyao · 2019-09-29T08:21:59Z

/bench

H-ZeX · 2019-09-29T08:22:36Z

/bench

H-ZeX · 2019-09-29T08:46:30Z

/bench/bench/bench/bench

Signed-off-by: H-ZeX <hzx20112012@gmail.com>

H-ZeX · 2019-10-10T03:44:38Z

/bench

H-ZeX · 2019-10-25T01:41:39Z

/bench

H-ZeX · 2019-10-25T02:05:15Z

/run-all-tests

H-ZeX · 2019-10-25T02:09:34Z

/bench

sre-bot · 2019-10-25T04:02:27Z

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
--- tidb: 4907685ed298c0efb4f09530cc7bf6c1e2843cd4
+++ tidb: c3aac21a6a9636298757c5160af81af2842476c8
tikv: 57aa7ba9edeb4132ba8bcef8ec916952ad343b0f
pd: 6e22c1c3cc7a266011d0a706a8cbfc2f6329af4a
================================================================================
oltp_update_non_index:
    * QPS: 4866.03 ± 0.22% (std=7.41) delta: 0.18% (p=0.410)
    * Latency p50: 26.30 ± 0.22% (std=0.04) delta: -0.18%
    * Latency p99: 42.45 ± 4.89% (std=1.46) delta: 0.01%
            
oltp_insert:
    * QPS: 4722.02 ± 0.03% (std=1.17) delta: 0.07% (p=0.472)
    * Latency p50: 27.10 ± 0.05% (std=0.01) delta: -0.06%
    * Latency p99: 49.45 ± 3.17% (std=1.16) delta: 0.49%
            
oltp_read_write:
    * QPS: 15596.42 ± 0.65% (std=66.71) delta: -0.62% (p=0.094)
    * Latency p50: 164.79 ± 0.29% (std=0.34) delta: 0.80%
    * Latency p99: 326.17 ± 5.31% (std=10.86) delta: 1.87%
            
oltp_update_index:
    * QPS: 4367.49 ± 0.22% (std=6.79) delta: 0.16% (p=0.556)
    * Latency p50: 29.31 ± 0.23% (std=0.05) delta: -0.16%
    * Latency p99: 55.59 ± 4.10% (std=1.49) delta: 2.00%
            
oltp_point_select:
    * QPS: 42509.41 ± 0.89% (std=237.23) delta: -0.67% (p=0.128)
    * Latency p50: 3.01 ± 1.00% (std=0.02) delta: 0.75%
    * Latency p99: 9.48 ± 0.90% (std=0.08) delta: 0.91%

XuHuaiyu

LGTM

XuHuaiyu · 2019-10-28T06:22:35Z

PTAL @qw4990

zz-jason

LGTM

sre-bot · 2019-10-28T11:14:37Z

Your auto merge job has been accepted, waiting for 12862

sre-bot · 2019-10-28T11:19:31Z

/run-all-tests

sre-bot · 2019-10-28T11:25:09Z

cherry pick to release-2.1 failed

sre-bot · 2019-10-28T11:25:42Z

cherry pick to release-3.1 failed

sre-bot · 2019-10-28T11:26:08Z

cherry pick to release-3.0 failed

sre-bot · 2020-04-07T10:24:07Z

It seems that, not for sure, we failed to cherry-pick this commit to release-3.0. Please comment '/run-cherry-picker' to try to trigger the cherry-picker if we did fail to cherry-pick this commit before. @XuHuaiyu PTAL.

XuHuaiyu · 2020-04-08T02:17:37Z

/run-cherry-picker

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot · 2020-04-08T02:19:18Z

cherry pick to release-3.0 in PR #16141

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot · 2020-04-08T02:22:00Z

cherry pick to release-3.1 in PR #16142

fix_column_null_bitmap_too_short

da42f1c

Signed-off-by: H-ZeX <hzx20112012@gmail.com>

fix ci failed

cc9682c

Signed-off-by: H-ZeX <hzx20112012@gmail.com>

H-ZeX changed the title ~~util: increase column's nullBitmap's capacity~~ util: increase column's nullBitmap's capacity and change the way to shorten a slice Sep 29, 2019

revert some change

a00865a

Signed-off-by: H-ZeX <hzx20112012@gmail.com>

H-ZeX force-pushed the fix_column_null_bitmap_too_short branch from 4d3ad5b to a00865a Compare October 10, 2019 02:42

H-ZeX added 2 commits October 10, 2019 10:44

revert some change

68e1e55

Signed-off-by: H-ZeX <hzx20112012@gmail.com>

Merge branch 'master' into fix_column_null_bitmap_too_short

cd30605

Merge branch 'master' into fix_column_null_bitmap_too_short

c3aac21

zz-jason requested review from qw4990 and XuHuaiyu October 25, 2019 01:58

H-ZeX added component/util needs-cherry-pick-2.1 labels Oct 25, 2019

H-ZeX changed the title ~~util: increase column's nullBitmap's capacity and change the way to shorten a slice~~ util: increase column's nullBitmap's capacity Oct 25, 2019

XuHuaiyu added the type/enhancement The issue or PR belongs to an enhancement. label Oct 28, 2019

XuHuaiyu reviewed Oct 28, 2019

View reviewed changes

XuHuaiyu added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 28, 2019

zz-jason approved these changes Oct 28, 2019

View reviewed changes

zz-jason added status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 28, 2019

Merge branch 'master' into fix_column_null_bitmap_too_short

694ca33

sre-bot merged commit b3ad90e into pingcap:master Oct 28, 2019

H-ZeX deleted the fix_column_null_bitmap_too_short branch October 28, 2019 14:43

H-ZeX mentioned this pull request Oct 28, 2019

util: increase column's nullBitmap's capacity (#12470) #12981

Merged

sre-bot pushed a commit that referenced this pull request Nov 1, 2019

util: increase column's nullBitmap's capacity (#12470) (#12981)

4a6c705

XiaTianliang pushed a commit to XiaTianliang/tidb that referenced this pull request Dec 21, 2019

util: increase column's nullBitmap's capacity (pingcap#12470)

8e0d9d5

XuHuaiyu added the needs-cherry-pick-3.1 label Apr 8, 2020

sre-bot pushed a commit to sre-bot/tidb that referenced this pull request Apr 8, 2020

cherry pick pingcap#12470 to release-3.0

1711060

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot mentioned this pull request Apr 8, 2020

util: increase column's nullBitmap's capacity (#12470) #16141

Merged

sre-bot pushed a commit to sre-bot/tidb that referenced this pull request Apr 8, 2020

cherry pick pingcap#12470 to release-3.1

3f7ad51

Signed-off-by: sre-bot <sre-bot@pingcap.com>

sre-bot mentioned this pull request Apr 8, 2020

util: increase column's nullBitmap's capacity (#12470) #16142

Merged

sre-bot added a commit that referenced this pull request Apr 9, 2020

util: increase column's nullBitmap's capacity (#12470) (#16141)

792ce6c

sre-bot added a commit that referenced this pull request Apr 9, 2020

util: increase column's nullBitmap's capacity (#12470) (#16142)

80924b8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

util: increase column's nullBitmap's capacity #12470

util: increase column's nullBitmap's capacity #12470

H-ZeX commented Sep 29, 2019 •

edited

Loading

winkyao commented Sep 29, 2019

codecov bot commented Sep 29, 2019 •

edited

Loading

winkyao commented Sep 29, 2019

H-ZeX commented Sep 29, 2019

H-ZeX commented Sep 29, 2019

H-ZeX commented Oct 10, 2019

H-ZeX commented Oct 25, 2019

H-ZeX commented Oct 25, 2019

H-ZeX commented Oct 25, 2019

sre-bot commented Oct 25, 2019

XuHuaiyu left a comment

XuHuaiyu commented Oct 28, 2019

zz-jason left a comment

sre-bot commented Oct 28, 2019

sre-bot commented Oct 28, 2019

sre-bot commented Oct 28, 2019

sre-bot commented Oct 28, 2019

sre-bot commented Oct 28, 2019

sre-bot commented Apr 7, 2020

XuHuaiyu commented Apr 8, 2020

sre-bot commented Apr 8, 2020

sre-bot commented Apr 8, 2020

util: increase column's nullBitmap's capacity #12470

util: increase column's nullBitmap's capacity #12470

Conversation

H-ZeX commented Sep 29, 2019 • edited Loading

What problem does this PR solve?

nullBitmap's cap

What is changed and how it works?

Check List

Side effects

Related changes

Release note

winkyao commented Sep 29, 2019

codecov bot commented Sep 29, 2019 • edited Loading

Codecov Report

winkyao commented Sep 29, 2019

H-ZeX commented Sep 29, 2019

H-ZeX commented Sep 29, 2019

H-ZeX commented Oct 10, 2019

H-ZeX commented Oct 25, 2019

H-ZeX commented Oct 25, 2019

H-ZeX commented Oct 25, 2019

sre-bot commented Oct 25, 2019

Benchmark Report

XuHuaiyu left a comment

Choose a reason for hiding this comment

XuHuaiyu commented Oct 28, 2019

zz-jason left a comment

Choose a reason for hiding this comment

sre-bot commented Oct 28, 2019

sre-bot commented Oct 28, 2019

sre-bot commented Oct 28, 2019

sre-bot commented Oct 28, 2019

sre-bot commented Oct 28, 2019

sre-bot commented Apr 7, 2020

XuHuaiyu commented Apr 8, 2020

sre-bot commented Apr 8, 2020

sre-bot commented Apr 8, 2020

H-ZeX commented Sep 29, 2019 •

edited

Loading

`nullBitmap`'s cap

codecov bot commented Sep 29, 2019 •

edited

Loading