-
Notifications
You must be signed in to change notification settings - Fork 891
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
List lexicographic comparator #11129
Merged
Merged
Changes from all commits
Commits
Show all changes
91 commits
Select commit
Hold shift + click to select a range
933c974
First commit
devavret a1636e5
testing and profiling deep single hierarchy struct
devavret d59f54c
Merge branch 'branch-22.02' into struct-row-comp
devavret 765dd8d
Merge branch 'branch-22.02' into struct-row-comp
devavret 3d21daf
Make the sandboxed test compile again
devavret 9f32e6b
Update my row_comparator with nullate
devavret 53d3c90
Merge branch 'branch-22.02' into struct-row-comp
devavret 022e2a4
Basic verticalization utility and experimental namespace
devavret 7fef643
clean up most of row operators that I didn't change.
devavret 930d8de
Sliced column test
devavret 0ecc4f8
column order and null precendence support
devavret ff36d2d
Manually managed stack
devavret cd0f938
New depth based method to avoid superimpose nulls
devavret 7b8e060
Put sort2 impl in separate TU
devavret 25eb237
Merge branch 'branch-22.04' into struct-row-comp
devavret d2937cf
Merge branch 'branch-22.04' into struct-row-comp
devavret d55c9c7
Move verticalization code to row_comparator.cpp
devavret 3bd749e
Owning row lex operator
devavret 613d664
merge fixes
devavret 2ef3ac7
Move struct logic out of main row loop and into element_relational_co…
devavret 5577431
pushing even more logic into element_relational_comparator
devavret f037bc0
More optimizations.
devavret 8c54a85
review changes
devavret 9d24a87
Checks to ensure tables can be compared
devavret a664c81
Super basic list lex working
devavret 1ebd877
list test expansion and cleanups.
devavret 3e6e9f4
Make struct comp work again
devavret facc031
List lex benchmark
devavret a19b2c3
Merge branch 'branch-22.08' into list-lex-comp
devavret 11bcf16
Add back code from old lex comparator that had list flattening
devavret 53f4418
Move list lex code to experimental header
devavret 0e528f0
get list lex working on code ported to exp header
devavret f545af0
Add null handling
devavret a4190a0
handle empty lists
devavret 9362b8d
Add sliced list test
devavret a7ec09b
Use progressive slicing to get leaf column
devavret d6ef822
Clean up old experiment files
devavret 5f0d36e
Turn dremel data raw pointers to spans
devavret 941b808
Replace bench with nvbench and fix destroyed dremel data issue
devavret 16e11cb
Merge branch 'branch-22.08' into list-lex-comp
devavret 2bd2b6b
More benchmark iterations
devavret 4be403c
merge pointers to dremel data into a view class
devavret 8cbd70c
Allow lhs and rhs dremel data
devavret 4640383
Merge branch 'branch-22.08' into list-lex-comp
devavret 3f43968
reduce test verbosity
devavret 4b233dc
Remove debug prints
devavret 6be6078
rename linked column header
devavret 8e4c870
Move dremel specific code out into spearate files
devavret ee13936
pass _comparator to elem comt
devavret 499a5bd
Remove lines that deal with dremel data as separate variables
devavret 0c3c12e
remove requirement to pass d_nullability and allow dremel_device_view…
devavret b62d0a2
Let get_dremel_data work without nullability
devavret a180bcc
Merge remote-tracking branch 'origin/branch-22.08' into list-lex-comp
vyasr 229ebe3
Update meta.yaml.
vyasr 926e7ab
Consolidate and augment descriptions of Dremel encoding.
vyasr d1cea06
Fix style.
vyasr 6030b7b
Remove unnecessary optionals around dremel_device_view.
vyasr fbb9dd3
Simplify list_lex_preprocess.
vyasr 25c22f9
Add some extra comments and docstrings.
vyasr 5305349
Add extensive comments explaining the list comparison algorithm.
vyasr 9b5f8c1
Reorder declarations for improved readability and logical consistency.
vyasr b334d19
Address open PR comments.
vyasr 1e31ac1
Enable previously disabled test.
vyasr 7c77616
Clean up comment.
vyasr 1f6b050
Address first round of PR comments.
vyasr c35a39a
Move dremel files to lists/detail.
vyasr b520e38
Fix header.
vyasr ff66bdb
Merge remote-tracking branch 'origin/branch-22.08' into list-lex-comp
vyasr 5a637d4
Merge remote-tracking branch 'origin/branch-22.10' into list-lex-comp
vyasr e8ebcc4
Try separating out primitive comparison.
vyasr 77c57bf
Revert "Try separating out primitive comparison."
vyasr 46f234f
Address most simple review comments.
vyasr 6d89799
Merge remote-tracking branch 'origin/branch-22.10' into list-lex-comp
vyasr 6b4ce40
Merge remote-tracking branch 'origin/branch-22.10' into list-lex-comp
vyasr b32205d
Update benchmark for new data generation API.
vyasr d578c8b
Add method to check for nested columns in a table_view.
vyasr 0671246
Template comparator on the presence of nested columns and propagate p…
vyasr 8c0ae93
Only define the list/struct overloads in the specialization that coul…
vyasr d285df9
Move the specialization to a completely separate class.
vyasr 31a9bfd
Revert "Move the specialization to a completely separate class."
vyasr 36cc5f3
Fix typo.
vyasr 8dd293a
Convert the Dremel members of the preprocessed_table to optionals.
vyasr 7b0ae58
Propagate optionals down to the element comparator.
vyasr be3ab5e
Revert "Propagate optionals down to the element comparator."
vyasr 4a735d3
Stop storing empty dremel views for non-list columns and use a thrust…
vyasr ab5a264
Some cleanup.
vyasr d58ad80
Remove unnecessary check.
vyasr 1f6b7ab
Address PR comments.
vyasr a1a9655
Revert "Remove unnecessary check."
vyasr f5cee47
Address remaining TODOs.
vyasr f7b671a
Fix typo.
vyasr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
/* | ||
* Copyright (c) 2022, NVIDIA CORPORATION. | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
#include <benchmarks/common/generate_input.hpp> | ||
#include <benchmarks/fixture/rmm_pool_raii.hpp> | ||
|
||
#include <cudf/detail/sorting.hpp> | ||
|
||
#include <nvbench/nvbench.cuh> | ||
|
||
void nvbench_sort_lists(nvbench::state& state) | ||
{ | ||
cudf::rmm_pool_raii pool_raii; | ||
|
||
const size_t size_bytes(state.get_int64("size_bytes")); | ||
const cudf::size_type depth{static_cast<cudf::size_type>(state.get_int64("depth"))}; | ||
auto const null_frequency{state.get_float64("null_frequency")}; | ||
|
||
data_profile table_profile; | ||
table_profile.set_distribution_params(cudf::type_id::LIST, distribution_id::UNIFORM, 0, 5); | ||
table_profile.set_list_depth(depth); | ||
table_profile.set_null_probability(null_frequency); | ||
auto const table = | ||
create_random_table({cudf::type_id::LIST}, table_size_bytes{size_bytes}, table_profile); | ||
|
||
state.exec(nvbench::exec_tag::sync, [&](nvbench::launch& launch) { | ||
rmm::cuda_stream_view stream_view{launch.get_stream()}; | ||
cudf::detail::sorted_order(*table, {}, {}, stream_view, rmm::mr::get_current_device_resource()); | ||
}); | ||
} | ||
|
||
NVBENCH_BENCH(nvbench_sort_lists) | ||
.set_name("sort_list") | ||
.add_int64_power_of_two_axis("size_bytes", {10, 18, 24, 28}) | ||
.add_int64_axis("depth", {1, 4}) | ||
.add_float64_axis("null_frequency", {0, 0.2}); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Previously I tried to use a lambda to simplify the code:
Unfortunately using such variadic template causes false compiler warnings. I want to bring it back here in case you guys may have a new idea for improving this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even if not, we can also reduce the code by half by writing a lambda for the block
lower_bound
/upper_bound
and calling it twice.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See #11667 for discussion on how we can improve this. The plan is to do that in a follow-up so that we don't hold up this feature any longer for refactoring-like tasks.