Expose pack/unpack API to Python

[charlesbluca]

Closes #7601

Adds a Python API for pack/unpack, so that we might be able to pack/unpack DataFrames in serialization:

• PackedColumns is a Python representation of the cudf::packed_columns struct containing the struct itself along with some Python metadata for the DataFrame being packed; supports Dask/pickle serialization
• pack() takes in a Table and returns a PackedColumns
• unpack() takes in a PackedColumns and returns a Table

cc @brandon-b-miller

[codecov]

Codecov Report

Merging #8153 (82c6aba) into branch-21.08 (1e53776) will increase coverage by 0.00%.
The diff coverage is 0.00%.

❗ Current head 82c6aba differs from pull request most recent head 3fa48c1. Consider uploading reports for the commit 3fa48c1 to get more accurate results

@@              Coverage Diff               @@
##           branch-21.08    #8153    +/-   ##
==============================================
  Coverage         10.62%   10.62%            
==============================================
  Files               109      109            
  Lines             18246    18627   +381     
==============================================
+ Hits               1938     1980    +42     
- Misses            16308    16647   +339     

Impacted Files	Coverage Δ
python/cudf/cudf/core/column/column.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/column/datetime.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/column/decimal.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/column/numerical.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/column/string.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/dataframe.py	0.00% <ø> (ø)
python/cudf/cudf/core/index.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/series.py	0.00% <0.00%> (ø)
python/cudf/cudf/core/window/rolling.py	0.00% <0.00%> (ø)
python/cudf/cudf/io/hdf.py	0.00% <0.00%> (ø)
... and 51 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e53776...3fa48c1. Read the comment docs.

[jakirkham]

Yep that makes sense. Figure we should test directly in case we change how pickling works later (not that I expect we will just want to make sure we have coverage in that event)

[jakirkham]

One last question, so IIUC we are simply adding the utilities for pack/unpack now, but not using them yet. Is that right?

Asking based on a discussion Ashwin and I had a while back about making pack/unpack configureable ( #5311 )

[charlesbluca]

Yeah, this PR doesn't do anything to change how dataframes are currently serialized - I imagined that the task of adding pack/unpack as an option for serialization would probably bring up a larger conversation about how we want to make cuDF configurable, which I thought would be better for a follow up PR

[jakirkham]

Yep that makes a lot of sense. Thanks for working on this btw 🙂

[charlesbluca]

rerun tests

[charlesbluca]

Not sure why this is failing on mypy in the style checks, seems to pass locally

[shwina]

The style checks should resolve with #8595.

[charlesbluca]

rerun tests

[jakirkham]

rerun tests

[charlesbluca]

rerun tests

[jakirkham]

rerun tests

[jakirkham]

@gpucibot merge

[jakirkham]

It appears we are seeing a testing failure related to this change

20:43:20 ___________________ ERROR collecting cudf/tests/test_pack.py ___________________
20:43:20 ImportError while importing test module '/workspace/python/cudf/cudf/tests/test_pack.py'.
20:43:20 Hint: make sure your test modules/packages have valid Python names.
20:43:20 Traceback:
20:43:20 /opt/conda/envs/rapids/lib/python3.7/importlib/__init__.py:127: in import_module
20:43:20     return _bootstrap._gcd_import(name[level:], package, level)
20:43:20 cudf/tests/test_pack.py:23: in <module>
20:43:20     from cudf.tests.utils import assert_eq
20:43:20 E   ModuleNotFoundError: No module named 'cudf.tests'

https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cudf/job/prb/job/cudf-gpu-test/CUDA=11.0,GPU_LABEL=gpu-a100,OS=centos7,PYTHON=3.7/2326/console

[charlesbluca]

Yup it looks like assert_eq was relocated to cudf.testing._utils

[charlesbluca]

@gpucibot merge