You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
KnownNotNull is used by Spark to tag an expression as known to not be null. A customer asked for this and we do not completely know the use case for it. I looked at the code for Spark 3.0.2, you probably want to check newer versions to see if there are other cases, but it appears to only be used with UDFs and in a few cases with conversions from a data set to data frames.
Essentially it is for a scala/java UDF that takes a primitive type as input, something that cannot be nullable. Then Spark inserts other processing around the UDF so it would never be called with a null. This is probably related to the native UDF stuff that @jlowe did, and is the only way I can think of that we would hit something like this in practice.
Is your feature request related to a problem? Please describe.
Implement KnownNotNull on GPU
Describe the solution you'd like
When KnownNotNull is part of the plan, the plan stays on the GPU
Describe alternatives you've considered
N/A
Additional context
The text was updated successfully, but these errors were encountered: