Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Implement KnownNotNull on the GPU #2562

Closed
sameerz opened this issue Jun 2, 2021 · 3 comments · Fixed by #2707
Closed

[FEA] Implement KnownNotNull on the GPU #2562

sameerz opened this issue Jun 2, 2021 · 3 comments · Fixed by #2707
Assignees
Labels
feature request New feature or request

Comments

@sameerz
Copy link
Collaborator

sameerz commented Jun 2, 2021

Is your feature request related to a problem? Please describe.
Implement KnownNotNull on GPU

Describe the solution you'd like
When KnownNotNull is part of the plan, the plan stays on the GPU

Describe alternatives you've considered
N/A

Additional context

@sameerz sameerz added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jun 2, 2021
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Jun 8, 2021
@firestarman
Copy link
Collaborator

firestarman commented Jun 9, 2021

@sameerz May I know the case/sql for this KnownNotNull ? Since I need a test case for it.

@revans2
Copy link
Collaborator

revans2 commented Jun 9, 2021

KnownNotNull is used by Spark to tag an expression as known to not be null. A customer asked for this and we do not completely know the use case for it. I looked at the code for Spark 3.0.2, you probably want to check newer versions to see if there are other cases, but it appears to only be used with UDFs and in a few cases with conversions from a data set to data frames.

Essentially it is for a scala/java UDF that takes a primitive type as input, something that cannot be nullable. Then Spark inserts other processing around the UDF so it would never be called with a null. This is probably related to the native UDF stuff that @jlowe did, and is the only way I can think of that we would hit something like this in practice.

@firestarman
Copy link
Collaborator

I tried several cases and got this expression in logical plan, but it will be removed after optimization.
I will check the the native UDF.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants