Migrate dask-cudf CudfEngine to leverage ArrowDatasetEngine

[rjzamora]

Closes #8656

Addresses the impending deprecation of the ArrowLegacyEngine (which dask-cudf currently depends on), by migrating the CudfEngine backend to the newer ArrowDatasetEngine.

TODO:

• Benchmark/check for any (significant) performance regressions (EDIT: pyarrow-deprecations in pyarrow-5 make this migration necessary IMO)

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.10@8e9f0aa). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head b1226f6 differs from pull request most recent head d672ab5. Consider uploading reports for the commit d672ab5 to get more accurate results

@@               Coverage Diff               @@
##             branch-21.10    #8871   +/-   ##
===============================================
  Coverage                ?   10.62%           
===============================================
  Files                   ?      116           
  Lines                   ?    18683           
  Branches                ?        0           
===============================================
  Hits                    ?     1986           
  Misses                  ?    16697           
  Partials                ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e9f0aa...d672ab5. Read the comment docs.

[quasiben]

A lot nicer with supported categorical column creation!

[rjzamora]

I just realized that the new change here was a bit inefficent. We were creating a column with the partition-based category repeated in every element, and then converting it to a categorical column. It makes a bit more sense to repeat the index of the partition-based category in every element, and build the categorical column directly.

[quasiben]

@rjzamora left one small comment (small typo) . I'll approve now though as i might not be available tomorrow morning

[rjzamora]

Thanks for the review @quasiben !

Note that I did revise a few small things since you approved the PR yesterday.

[galipremsagar]

rerun tests

[galipremsagar]

@gpucibot merge