You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe. #1393 added support for users supplying an alternate RAPIDS implementation of a Hive UDF, and that works well for queries that are already in SQL. However if the query is written against the DataFrame API and using a Scala UDF (seen in the Catalyst plan as ScalaUDF) then there isn't an option to provide a RAPIDS alternative implementation in the UDF. Spark wraps the user's code in at least one layer of lambda functions, so it's tricky to get access to the user's original class that implements their UDF code to see if it implements the RapidsUDF interface.
Describe the solution you'd like
Ideally the RAPIDS Accelerator plugin would be able to automatically identify whether the user code behind a ScalaUDF instance implements the RapidsUDF interface and therefore has a RAPIDS-accelerated implementation. Due to the use of wrapping lambdas, we may need to examine the bytecode and 'peel off' the lambda layers (if this is possible).
Describe alternatives you've considered
We could provide a separate method, specific to the RAPIDS Accelerator plugin, where users could register their UDFs, but that has two main drawbacks:
It requires the users to modify their code to additionally register the UDF
We'd still have to match up the code found in a ScalaUDF instance with the code registered via the separate interface.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
#1393 added support for users supplying an alternate RAPIDS implementation of a Hive UDF, and that works well for queries that are already in SQL. However if the query is written against the DataFrame API and using a Scala UDF (seen in the Catalyst plan as
ScalaUDF
) then there isn't an option to provide a RAPIDS alternative implementation in the UDF. Spark wraps the user's code in at least one layer of lambda functions, so it's tricky to get access to the user's original class that implements their UDF code to see if it implements theRapidsUDF
interface.Describe the solution you'd like
Ideally the RAPIDS Accelerator plugin would be able to automatically identify whether the user code behind a
ScalaUDF
instance implements theRapidsUDF
interface and therefore has a RAPIDS-accelerated implementation. Due to the use of wrapping lambdas, we may need to examine the bytecode and 'peel off' the lambda layers (if this is possible).Describe alternatives you've considered
We could provide a separate method, specific to the RAPIDS Accelerator plugin, where users could register their UDFs, but that has two main drawbacks:
ScalaUDF
instance with the code registered via the separate interface.The text was updated successfully, but these errors were encountered: