[BUG] cuDF cannot create list of struct dataframe using dict or from pandas

[devavret]

cuDF loses the field names of the struct inside a list when creating from a dict:

In [1]: import cudf

In [2]: import pandas as pd

In [3]: cudf.DataFrame({
   ...:             "family": [
   ...:                 [None, {"human?": True, "deets": {"weight": 2.4, "age": 27}}],
   ...:                 [
   ...:                     {"human?": None, "deets": {"weight": 5.3, "age": 25}},
   ...:                     {"human?": False, "deets": {"weight": 8.0, "age": 31}},
   ...:                     {"human?": False, "deets": None},
   ...:                 ],
   ...:                 [],
   ...:                 [{"human?": None, "deets": {"weight": 6.9, "age": None}}],
   ...:             ]
   ...:         })
Out[3]: 
                                              family
0    [None, {'0': {'0': 27.0, '1': 2.4}, '1': True}]
1  [{'0': {'0': 25.0, '1': 5.3}, '1': None}, {'0'...
2                                                 []
3          [{'0': {'0': None, '1': 6.9}, '1': None}]

or from pandas:

In [4]: pdf = pd.DataFrame({
   ...:             "family": [
   ...:                 [None, {"human?": True, "deets": {"weight": 2.4, "age": 27}}],
   ...:                 [
   ...:                     {"human?": None, "deets": {"weight": 5.3, "age": 25}},
   ...:                     {"human?": False, "deets": {"weight": 8.0, "age": 31}},
   ...:                     {"human?": False, "deets": None},
   ...:                 ],
   ...:                 [],
   ...:                 [{"human?": None, "deets": {"weight": 6.9, "age": None}}],
   ...:             ]
   ...:         })

In [5]: cudf.from_pandas(pdf)
Out[5]: 
                                              family
0    [None, {'0': {'0': 27.0, '1': 2.4}, '1': True}]
1  [{'0': {'0': 25.0, '1': 5.3}, '1': None}, {'0'...
2                                                 []
3          [{'0': {'0': None, '1': 6.9}, '1': None}]

The expected dataframe has field names:

In [6]: pdf
Out[6]: 
                                              family
0  [None, {'human?': True, 'deets': {'weight': 2....
1  [{'human?': None, 'deets': {'weight': 5.3, 'ag...
2                                                 []
3  [{'human?': None, 'deets': {'weight': 6.9, 'ag...

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[isVoid]

RCA: the issue is that during roundtrip to libcudf, field name information is lost, and column index is used to construct the field name. (Note that non-nested struct column preserves the field name because it goes a different code path)

cudf/python/cudf/cudf/_lib/types.pyx

Line 189 in 667b9bc

str(i): dtype_from_column_view(cv.child(i))

To resolve this, cudf need to pass along the field names from pyarrow column to reconstructed cudf column.

[isVoid]

RCA Update: synced with @shwina offline, inside cudf type system there is column._copy_type_metadata that handles field name transfer. However, when constructing a cudf column from pyarrow, there isn't a function that handles copying type from pyarrow array. The plan forward is to do type copy upon construction in from_arrow.