add Xorfilter #324
Closed
add Xorfilter #324
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chaser-ch
reviewed
Aug 24, 2021
| LIB_LOG(WARN, "the number of keys is not equal to the size_, ", K_(nbit), K(ret)); | ||
| return ret; | ||
| } | ||
|
|
| uint32_t* counting = (uint32_t*)tmp_allocator.alloc(sizeof(uint32_t) * array_length_); | ||
| uint64_t* buckets = (uint64_t*)tmp_allocator.alloc(sizeof(uint64_t) * array_length_); | ||
| int* queue = (int*)tmp_allocator.alloc(sizeof(int) * array_length_); | ||
|
|
There was a problem hiding this comment.
alloc once and assign to each pointer?
| int max_try = 20; | ||
| while (max_try-- > 0) { | ||
| seed_++; | ||
| memset(stack_hash, 0, sizeof(uint64_t) * size_); |
There was a problem hiding this comment.
alloc once and memset once
| if (key_pos_ >= size_) { | ||
| ret = OB_SIZE_OVERFLOW; | ||
| STORAGE_LOG(WARN, "rowkeys is already full", K(ret)); | ||
| } else if (OB_FAIL(rowkey.deep_copy(rowkeys_[key_pos_], *allocator))) { |
There was a problem hiding this comment.
we can also support insert single rowkey for xor filter builder, then the big space to keep all the rowkeys is no longer needed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add support for xorfilter.
Why are the changes needed?
xorfilter is smaller and faster than bloomfilter, but the build time is slower.
Will break the compatibility? How if so?
No.
Does this PR introduce any user-facing change?
Yes. The user can decide to use bloomfilter or xorfilter by adjusting the parameters
is_xorusing commandALTER SYSTEM SET is_xor = XXX(True or False)How was this patch tested?
unittest(test_bloom_filter_cache.cpp) and sysbench.
Checklist