[BUG] Unable to read parquet with cuDF which are written with ParquetWriter

[bschifferer]

Describe the bug
I write multiple dataframes with ParquetWriter to a single parquet file. The reason is, that convert tfrecords to parquet files. tfrecords could be large to load all into memory. Therefore, I need to write it into chunks to a parquet file.
When I try to load the parquet file with cuDF, I get following error:

ValueError: Length mismatch: Expected axis has 20 elements, new values have 5 elements

However, I am able to read the same parquet file with pandas.

Steps/Code to reproduce bug
cudf version: '21.06.01+0.g101fc0fda4.dirty'

import cudf

from cudf.io.parquet import ParquetWriter

df = cudf.DataFrame({'col1': [0,1,2,3,4]})

w = ParquetWriter('test1' + ".parquet")
w.write_table(df)
w.write_table(df)
w.write_table(df)
w.write_table(df)
w.close()

df1 = cudf.read_parquet('test1.parquet')

[benfred]

I also see this on cudf v21.08.02

[beckernick]

I am able to read the file, but we only get 5 records in our index (we expect 10 -- 5 per call to write_table)

import cudf
import pandas as pd
from cudf.io.parquet import ParquetWriter
​
df = cudf.DataFrame({'col1': [0,1,2,3,4]})
​
w = ParquetWriter('test1' + ".parquet")
w.write_table(df)
w.write_table(df)
w.close()
​
df1 = cudf.read_parquet('test1.parquet')
len(df1), len(pd.read_parquet("test1.parquet"))
(5, 10)

Our underlying issue appears to be that we construct a RangeIndex from 0 to 5, rather than 0 to 10. We have the full buffer of data.

df1 = cudf.read_parquet('test1.parquet')
df1.col1._column
<cudf.core.column.numerical.NumericalColumn object at 0x7fc0010367c0>
[
  0,
  1,
  2,
  3,
  4,
  0,
  1,
  2,
  3,
  4
]
dtype: int64

[devavret]

Seems to be a duplicate of #7011

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.