-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move pandas_udf functions into the tests functions #926
Conversation
compile when skipped
@firestarman @shotai any chance you have env setup you can make sure the test still works when config specified? |
build |
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
6ee7209
to
10bd6d6
Compare
Thanks a lot for finding this. One little issue is the two udfs Something like: firestarman@900a91d @cudf_udf
def test_window(enable_cudf_udf):
+ @pandas_udf("int")
+ def _sum_cpu_func(v: pd.Series) -> int:
+ return v.sum()
+
+ @pandas_udf("integer")
+ def _sum_gpu_func(v: pd.Series) -> int:
+ import cudf
+ gpu_series = cudf.Series(v)
+ return gpu_series.sum()
+
def cpu_run(spark):
df = _create_df(spark)
w = Window.partitionBy('id').rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)
return df.withColumn('sum_v', _sum_cpu_func('v').over(w)).collect() |
With the change above, i can get cudf udf tests passed locally. |
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
@firestarman thanks for the review, I added them to test_window. I realize this duplicates the functions so if there is a better way you can come up with not to duplicate the functions that is more ideal. I tried putting them in a class but that still compiled it. |
build |
1 similar comment
build |
* Move pandas_udf functions into the tests functions so they don't try to compile when skipped * put back enable hive Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Add missing functions to test_window Signed-off-by: Thomas Graves <tgraves@nvidia.com>
* Move pandas_udf functions into the tests functions so they don't try to compile when skipped * put back enable hive Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Add missing functions to test_window Signed-off-by: Thomas Graves <tgraves@nvidia.com>
* Move pandas_udf functions into the tests functions so they don't try to compile when skipped * put back enable hive Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Add missing functions to test_window Signed-off-by: Thomas Graves <tgraves@nvidia.com>
…IDIA#926) Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
Move pandas_udf functions into the tests functions so they don't try to compile when skipped.
Fixes #922
Note I can now run tests without this failing, but I haven't run the test to make sure it passes as I don't have the env setup to do it.