Enable backend dispatching for Dask-DataFrame creation #11920

rjzamora · 2022-10-14T13:44:37Z

Description

This PR depends on dask/dask#9475 (Now Merged)

After dask#9475, external libraries are now able to implement (and expose) their own DataFrameBackendEntrypoint definitions to specify custom creation functions for DataFrame collections. This PR introduces the CudfBackendEntrypoint class to create dask_cudf.DataFrame collections using the dask.dataframe API. By installing dask_cudf with this entrypoint definition in place, you get the following behavior in dask.dataframe:

import dask.dataframe as dd
import dask

# Tell Dask that you want to create DataFrame collections
# with the "cudf" backend (for supported creation functions).
# This can also be used in a context, or set in a yaml file
dask.config.set({"dataframe.backend": "cudf"})

ddf = dd.from_dict({"a": range(10)}, npartitions=2)
type(ddf)  # dask_cudf.core.DataFrame

Note that the code snippet above does not require an explicit import of cudf or dask_cudf. The following creation functions will support backend dispatching after dask#9475:

from_dict
read_paquet
read_json
read_orc
read_csv
read_hdf

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

…point name

codecov · 2022-10-14T16:46:49Z

Codecov Report

Base: 87.40% // Head: 88.13% // Increases project coverage by +0.72% 🎉

Coverage data is based on head (720e702) compared to base (f72c4ce).
Patch coverage: 89.92% of modified lines in pull request are covered.

❗ Current head 720e702 differs from pull request most recent head 3719558. Consider uploading reports for the commit 3719558 to get more accurate results

Additional details and impacted files

@@               Coverage Diff                @@
##           branch-22.12   #11920      +/-   ##
================================================
+ Coverage         87.40%   88.13%   +0.72%     
================================================
  Files               133      133              
  Lines             21833    21987     +154     
================================================
+ Hits              19084    19379     +295     
+ Misses             2749     2608     -141

Impacted Files	Coverage Δ
python/cudf/cudf/core/dataframe.py	`93.77% <ø> (ø)`
python/cudf/cudf/core/indexed_frame.py	`92.03% <ø> (ø)`
python/cudf/cudf/core/udf/__init__.py	`97.05% <ø> (+47.05%)`	⬆️
python/cudf/cudf/io/orc.py	`92.94% <ø> (-0.09%)`	⬇️
python/cudf/cudf/testing/dataset_generator.py	`72.83% <ø> (-0.42%)`	⬇️
...thon/dask_cudf/dask_cudf/tests/test_distributed.py	`18.86% <ø> (+4.94%)`	⬆️
python/cudf/cudf/core/_base_index.py	`82.20% <43.75%> (-3.35%)`	⬇️
python/cudf/cudf/io/text.py	`91.66% <66.66%> (-8.34%)`	⬇️
python/strings_udf/strings_udf/__init__.py	`84.31% <76.00%> (-12.57%)`	⬇️
python/dask_cudf/dask_cudf/backends.py	`84.90% <82.92%> (-0.37%)`	⬇️
... and 27 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

python/dask_cudf/dask_cudf/core.py

galipremsagar

Overall looks good Rick, few comments before this is ready to merge.

galipremsagar · 2022-10-17T19:06:50Z

python/dask_cudf/dask_cudf/backends.py

+        def from_dict(data, npartitions, orient="columns", **kwargs):
+            if orient != "columns":
+                raise ValueError(f"orient={orient} is not supported")
+            return dd.from_pandas(


Is from_pandas required here because we don't support cudf.DataFrame.from_dict API yet? If so, can we add a todo here to change this after #11934 is resolved?

Is from_pandas required here because we don't support cudf.DataFrame.from_dict API yet?

Yes and no - We should certainly use cudf.from_dict when it is supported (I'll add a TODO). However, I'll change the dd.from_pandas code to dask_cudf.from_cudf for clarity (from_cudf is just a cudf-friendly alias for from_pandas).

python/dask_cudf/dask_cudf/core.py

galipremsagar

Thanks @rjzamora !

wence-

Does the entrypoint need to be added to setup.py/cfg or similar?

python/dask_cudf/dask_cudf/backends.py

rjzamora · 2022-10-18T15:11:44Z

Does the entrypoint need to be added to setup.py/cfg or similar?

Correct - The entrypoint is defined in dask_cudf's setup.cfg file.

wence- · 2022-10-18T15:16:36Z

Does the entrypoint need to be added to setup.py/cfg or similar?

Correct - The entrypoint is defined in dask_cudf's setup.cfg file.

Ah, somehow I missed that this had already been done back in March

rjzamora · 2022-10-20T13:20:17Z

@gpucibot merge

rjzamora added 14 commits March 8, 2022 10:49

define and register CudfBackendEntrypoint entrypoint

5c6b86f

formatting

2001c29

Merge remote-tracking branch 'upstream/branch-22.10' into backend-class

a4e9292

tweak class names

9afc190

move fallback into backend entrypoints and revert to DaskBackendEntry…

e4e3639

…point name

align CudfBackendEntrypoint with latest backend-class branch in dask

fbaa4ee

fix bug in CudfBackendEntrypoint

8fe6ca8

remove timeseries dispatch for now

17919bd

define from_dict

81f8c36

fix DaskDataFrameBackendEntrypoint import

c7b7888

fix backend typo

784dc38

Merge remote-tracking branch 'upstream/branch-22.10' into backend-class

1a72c55

Merge remote-tracking branch 'upstream/branch-22.12' into backend-class

545a1a2

update entrypoint path

f4704f4

rjzamora added 2 - In Progress Currently a work in progress dask Dask issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change helps: Dask labels Oct 14, 2022

github-actions bot added the Python Affects Python cuDF API. label Oct 14, 2022

rjzamora added 5 commits October 14, 2022 06:51

cleanup - still need to add specific backend-dispatching tessts

76118a8

Merge remote-tracking branch 'upstream/branch-22.12' into backend-class

3bb4701

formatting

964e53a

remove extra line

0f7d2f7

more formatting

41bc7ce

rjzamora added 2 commits October 14, 2022 14:56

remove redundant entrypoint info

e4313e9

add test coverage

22e53d3

rjzamora marked this pull request as ready for review October 17, 2022 15:34

rjzamora requested a review from a team as a code owner October 17, 2022 15:34

beckernick mentioned this pull request Oct 17, 2022

[DOC] Update 10 minutes to cudf and dask-cudf for clarity on compute vs persist #11931

Closed

galipremsagar reviewed Oct 17, 2022

View reviewed changes

python/dask_cudf/dask_cudf/core.py Outdated Show resolved Hide resolved

use DASK_BACKEND_SUPPORT

540f469

galipremsagar reviewed Oct 17, 2022

View reviewed changes

code review

7fb5272

galipremsagar approved these changes Oct 17, 2022

View reviewed changes

wence- reviewed Oct 18, 2022

View reviewed changes

python/dask_cudf/dask_cudf/backends.py Outdated Show resolved Hide resolved

python/dask_cudf/dask_cudf/backends.py Show resolved Hide resolved

python/dask_cudf/dask_cudf/backends.py Show resolved Hide resolved

rjzamora added 2 commits October 18, 2022 06:42

improve documentation

8d0926c

typo fix

672024a

adding from_dask_array and from_array preemptively

ffc444a

rjzamora marked this pull request as draft October 19, 2022 13:23

Roll back from_dask_array and from_array for now

3719558

rjzamora marked this pull request as ready for review October 19, 2022 17:09

rjzamora added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 2 - In Progress Currently a work in progress labels Oct 20, 2022

rapids-bot bot merged commit 416d4d5 into rapidsai:branch-22.12 Oct 20, 2022

rjzamora deleted the backend-class branch October 20, 2022 13:20

beckernick mentioned this pull request Oct 24, 2022

[BUG] dask_cudf read_csv does not support dask.dataframe's blocksize argument (vs. chunksize) #11847

Closed

quasiben mentioned this pull request Nov 30, 2022

[DOC] RAPIDS 22.12 Release Blog Outline #12057

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable backend dispatching for Dask-DataFrame creation #11920

Enable backend dispatching for Dask-DataFrame creation #11920

rjzamora commented Oct 14, 2022 •

edited

Loading

codecov bot commented Oct 14, 2022 •

edited

Loading

galipremsagar left a comment

galipremsagar Oct 17, 2022

rjzamora Oct 17, 2022

galipremsagar left a comment

wence- left a comment

rjzamora commented Oct 18, 2022

wence- commented Oct 18, 2022

rjzamora commented Oct 20, 2022

Enable backend dispatching for Dask-DataFrame creation #11920

Enable backend dispatching for Dask-DataFrame creation #11920

Conversation

rjzamora commented Oct 14, 2022 • edited Loading

Description

Checklist

codecov bot commented Oct 14, 2022 • edited Loading

Codecov Report

galipremsagar left a comment

Choose a reason for hiding this comment

galipremsagar Oct 17, 2022

Choose a reason for hiding this comment

rjzamora Oct 17, 2022

Choose a reason for hiding this comment

galipremsagar left a comment

Choose a reason for hiding this comment

wence- left a comment

Choose a reason for hiding this comment

rjzamora commented Oct 18, 2022

wence- commented Oct 18, 2022

rjzamora commented Oct 20, 2022

rjzamora commented Oct 14, 2022 •

edited

Loading

codecov bot commented Oct 14, 2022 •

edited

Loading