Support contains() on lists of primitives #7039
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mythrocks
added
4 - Needs Review
Update CHANGELOG, + headers yaml
davidwendt
requested changes
Dec 18, 2020
Codecov Report
@@ Coverage Diff @@
## branch-0.18 #7039 +/- ##
===============================================
+ Coverage 82.09% 82.20% +0.11%
===============================================
Files 97 98 +1
Lines 16474 16692 +218
===============================================
+ Hits 13524 13722 +198
- Misses 2950 2970 +20
Continue to review full report at Codecov.
|
Review feedback: 1. Optimized case where skey is null 2. Rephrased nested ternary if.
davidwendt
reviewed
Dec 18, 2020
Review: 3. Construct mutable output view only if required.
|
Full disclosure: I should probably also implement the I'll follow the advice here for that as well. |
mythrocks
commented
Dec 18, 2020
Fix null-mask for non-null list rows containing null elements.
mythrocks
commented
Dec 19, 2020
Further correction for null_mask
Further correction for null_mask
Added support for skey columns (instead of just scalars).
devavret
reviewed
Dec 28, 2020
There was a problem hiding this comment.
- Seems odd to have an API that only works on 1 level lists. Just for the sake of completeness, I'd've tried to make this generic.
- The only requirement for supported types seems to be an equality operator. This can work for chrono types. Not sure why they're implicitly left out.
ajschmidt8
approved these changes
Dec 30, 2020
karthikeyann
requested changes
Jan 23, 2021
karthikeyann
requested changes
Jan 25, 2021
karthikeyann
approved these changes
Jan 25, 2021
|
Hey, thanks for the review, @karthikeyann. I have rebased things. Adding the |
mythrocks
added
the
5 - Ready to Merge
|
(Argh. @karthikeyann, please ignore the request for re-review. That was by accident.) |
|
@gpucibot merge |
rapids-bot bot
pushed a commit
that referenced
this pull request
Jan 27, 2021
Adds JNI and Java side bindings for `list_contains` that is being added as part of #7039. Authors: - Kuhu Shukla (@kuhushukla) Approvers: - Robert (Bobby) Evans (@revans2) - MithunR (@mythrocks) URL: #7125
This was referenced Jan 30, 2021
mythrocks
added a commit
to mythrocks/cudf
that referenced
this pull request
Oct 23, 2021
`lists::contains()` (introduced in rapidsai#7039) returns a `BOOL8` column, indicating whether the specified search_key(s) exist at all in each corresponding list row of an input LIST column. It does not return the actual position. This commit introduces `lists::index_of()`, to return the INT32 positions of the specified search_key(s) in a LIST column. The search keys may be searched for using either `FIND_FIRST` (which finds the position of the first occurrence), or `FIND_LAST` (which finds the last occurrence). Both column_view and scalar search keys are supported. As with `lists::contains()`, nested types are not supported as search keys is `lists::index_of()`. If the search_key cannot be found, that output row is set to `-1`. Additionally, the row `output[i]` is set to null if: 1. The search_key(scalar) or search_keys[i](column_view) is null. 2. The list row `lists[i]` is null In all other cases, `output[i]` should contain a non-negative value.
mythrocks
added a commit
to mythrocks/cudf
that referenced
this pull request
Nov 5, 2021
`lists::contains()` (introduced in rapidsai#7039) returns a `BOOL8` column, indicating whether the specified search_key(s) exist at all in each corresponding list row of an input LIST column. It does not return the actual position. This commit introduces `lists::index_of()`, to return the INT32 positions of the specified search_key(s) in a LIST column. The search keys may be searched for using either `FIND_FIRST` (which finds the position of the first occurrence), or `FIND_LAST` (which finds the last occurrence). Both column_view and scalar search keys are supported. As with `lists::contains()`, nested types are not supported as search keys is `lists::index_of()`. If the search_key cannot be found, that output row is set to `-1`. Additionally, the row `output[i]` is set to null if: 1. The search_key(scalar) or search_keys[i](column_view) is null. 2. The list row `lists[i]` is null In all other cases, `output[i]` should contain a non-negative value.
mythrocks
added a commit
to mythrocks/cudf
that referenced
this pull request
Nov 5, 2021
`lists::contains()` (introduced in rapidsai#7039) returns a `BOOL8` column, indicating whether the specified search_key(s) exist at all in each corresponding list row of an input LIST column. It does not return the actual position. This commit introduces `lists::index_of()`, to return the INT32 positions of the specified search_key(s) in a LIST column. The search keys may be searched for using either `FIND_FIRST` (which finds the position of the first occurrence), or `FIND_LAST` (which finds the last occurrence). Both column_view and scalar search keys are supported. As with `lists::contains()`, nested types are not supported as search keys is `lists::index_of()`. If the search_key cannot be found, that output row is set to `-1`. Additionally, the row `output[i]` is set to null if: 1. The search_key(scalar) or search_keys[i](column_view) is null. 2. The list row `lists[i]` is null In all other cases, `output[i]` should contain a non-negative value.
mythrocks
added a commit
to mythrocks/cudf
that referenced
this pull request
Nov 23, 2021
`lists::contains()` (introduced in rapidsai#7039) returns a `BOOL8` column, indicating whether the specified search_key(s) exist at all in each corresponding list row of an input LIST column. It does not return the actual position. This commit introduces `lists::index_of()`, to return the INT32 positions of the specified search_key(s) in a LIST column. The search keys may be searched for using either `FIND_FIRST` (which finds the position of the first occurrence), or `FIND_LAST` (which finds the last occurrence). Both column_view and scalar search keys are supported. As with `lists::contains()`, nested types are not supported as search keys is `lists::index_of()`. If the search_key cannot be found, that output row is set to `-1`. Additionally, the row `output[i]` is set to null if: 1. The search_key(scalar) or search_keys[i](column_view) is null. 2. The list row `lists[i]` is null In all other cases, `output[i]` should contain a non-negative value.
rapids-bot bot
pushed a commit
that referenced
this pull request
Dec 20, 2021
Fixes #9164. ### Prelude `lists::contains()` (introduced in #7039) returns a `BOOL8` column, indicating whether the specified search_key(s) exist at all in each corresponding list row of an input LIST column. It does not return the actual position. ### `index_of()` This commit introduces `lists::index_of()`, to return the INT32 positions of the specified search_key(s) in a LIST column. The search keys may be searched for using either `FIND_FIRST` (which finds the position of the first occurrence), or `FIND_LAST` (which finds the last occurrence). Both column_view and scalar search keys are supported. As with `lists::contains()`, nested types are not supported as search keys in `lists::index_of()`. If the search_key cannot be found, that output row is set to `-1`. Additionally, the row `output[i]` is set to null if: 1. The `search_key`(scalar) or `search_keys[i]`(column_view) is null. 2. The list row `lists[i]` is null In all other cases, `output[i]` should contain a non-negative value. ### Semantic changes for `lists::contains()` This commit also modifies the semantics of `lists::contains()`: it will now return nulls only for the following cases: 1. The `search_key`(scalar) or `search_keys[i]`(column_view) is null. 2. The list row `lists[i]` is null In all other cases, a non-null bool is returned. Specifically `lists::contains()` no longer conforms to SQL semantics of returning `NULL` for list rows that don't contain the search key, while simultaneously containing nulls. In this case, `false` is returned. ### `lists::contains_null_elements()` A new function has been introduced to check if each list row contains null elements. The semantics are similar to `lists::contains()`, in that the column returned is BOOL8 typed: 1. If even 1 element in a list row is null, the returned row is `true`. 2. If no element is null, the returned row is `false`. 3. If the list row is null, the returned row is `null`. 4. If the list row is empty, the returned row is `false`. The current implementation is an inefficient placeholder, to be replaced once (#9588) is available. It is included here to reconstruct the SQL semantics dropped from `lists::contains()`. Authors: - MithunR (https://github.com/mythrocks) Approvers: - GALI PREM SAGAR (https://github.com/galipremsagar) - Jason Lowe (https://github.com/jlowe) - Mark Harris (https://github.com/harrism) - Conor Hoekstra (https://github.com/codereport) URL: #9510
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #6944.
This commit adds a method (
contains()) to check whether each row of aLISTcolumn contains the scalar value specified as an argument. The operation returns aBOOL8column (with as many rows as the inputLIST), each row indicatingtrueif the value is found,falseif not.Output
column[i]is set to null if even one of the following holds true (in line with the semantics ofarray_contains()in SQL):skeyis nulllists[i]is nulllists[i]contains even one null, andlists[i]does not contain the search key.This implementation currently supports the operation on lists of numerics or strings.