[BUG] Unstack function changes column names to numeric format

[BenikaHall]

Describe the bug
After applying the unstack function, the variable names change to numeric format.

Steps/Code to reproduce bug

def get_df(length, num_cols, num_months, acc_offset):
    cols = [ 'var_{}'.format(i) for i in range(num_cols)]
    df = cudf.DataFrame({col: cupy.random.rand(length * num_months) for col in cols})
    df['acc_id'] = cupy.repeat(cupy.arange(length), num_months) + acc_offset
    months = cupy.repeat(cupy.arange(length), num_months, axis=0).reshape(length, num_months)
    cupy.random.shuffle(months)
    df['month_id'] = months.T.flatten()
    return df

num_cols = 10
acc_len = 20
num_partitions = 4
num_months = 24

df = get_df(acc_len, num_cols, num_months, 0)

cols = [ 'var_{}'.format(i) for i in range(num_cols)]
unpivot = cudf.melt(df, id_vars=['acc_id','month_id'], value_vars=cols, var_name='name')

sorted_df = unpivot.sort_values(['acc_id', 'month_id'])

sorted_df.set_index(['acc_id', 'month_id', 'name']).unstack('name')

Expected behavior
After applying the unstack function, the column names should be preserved and not changed to 0,1,2,3,4,....

Environment overview (please complete the following information)

• Environment location: Docker
• Method of cuDF install: Rapids 0.17 container from NGC
  • If method of install is [Docker], provide docker pull & docker run commands used

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Additional context
Add any other context about the problem here.

[BenikaHall]

It appears that the dtype of the column we're trying to unstack is "category." It may be worth investigating why the "unstack" function changes the variable names for "category" dtypes. To resolve this, I set the column we wanted to unstack astype "object" and it worked seamlessly.

Adjusting the code shown in the initial comment, it now works as follows.

def get_df(length, num_cols, num_months, acc_offset):
    cols = [ 'var_{}'.format(i) for i in range(num_cols)]
    df = cudf.DataFrame({col: cupy.random.rand(length * num_months) for col in cols})
    df['acc_id'] = cupy.repeat(cupy.arange(length), num_months) + acc_offset
    months = cupy.repeat(cupy.arange(length), num_months, axis=0).reshape(length, num_months)
    cupy.random.shuffle(months)
    df['month_id'] = months.T.flatten()
    return df

num_cols = 10
acc_len = 20
num_partitions = 4
num_months = 24

df = get_df(acc_len, num_cols, num_months, 0)

cols = [ 'var_{}'.format(i) for i in range(num_cols)]
unpivot = cudf.melt(df, id_vars=['acc_id','month_id'], value_vars=cols, var_name='name')

sorted_df = unpivot.sort_values(['acc_id', 'month_id'])

sorted_df['name'] = sorted_df['name'].astype('object')
sorted_df.set_index(['acc_id', 'month_id', 'name']).unstack('name')
sorted_df

[shwina]

Thanks for reporting. I hope you don't mind that I reopened this issue because this certainly sound like there's still a bug somewhere. One shouldn't have to convert their categorical data to object (i.e. strings in cuDF) for unstack to work correctly.

[shwina]

I was able to track this down to a bug in the Frame._encode function:

cudf/python/cudf/cudf/core/frame.py

Lines 3493 to 3496 in aa72df7

	def _encode(self):
	keys, indices = libcudf.transform.table_encode(self)
	keys = self.__class__._from_table(keys)
	return keys, indices

Before returning keys, we should call keys._copy_type_metadata(self) to copy the category information from self onto keys.

[BenikaHall]

That's very true. Thank you for reopening @shwina. I'll follow up on this as well.

[shwina]

@BenikaHall - if you (or anyone else) would like to contribute a fix here, it should be relatively easy 🙂

In addition to the fix above, we'll need a couple of tests for this case in test_reshape.py.

[BenikaHall]

I'll see if I have some time next week to get a few tests done and a PR submitted. Thanks again @shwina.

[esnvidia]

Slightly related - calling unstack on a Series:

import cudf
import numpy as np
gdf = cudf.DataFrame(data=[[i for i in range(1990,2021) for j in range(500)],\
                     np.random.randint(0,3, size=(2021-1990)*500)]).T
gdf.columns=['year', 'use chip']
gdf[['year', 'use chip']].groupby(['year', 'use chip']).size().unstack(fill_value=0) # Fails
AttributeError: 'Series' object has no attribute 'unstack'
gdf[['year', 'use chip']].groupby(['year', 'use chip']).size().to_pandas().unstack(fill_value=0)  # success
cudf.__version__ --> 0.18.1

working around by calling to_frame() instead of to_pandas() causes MultiIndex to be created:

gdf[['year', 'use chip']].groupby(['year', 'use chip']).size().to_frame().unstack().columns
MultiIndex([(0, 0),
            (0, 1),
            (0, 2)],
           names=[None, 'use chip'])
gdf[['year', 'use chip']].groupby(['year', 'use chip']).size().to_pandas().unstack().columns
Int64Index([0, 1, 2], dtype='int64', name='use chip')