[DOC] Add libcudf policy that list columns must be sanitized

[nvdbaranec]

I found a case where the parquet writer seems to be incorrectly writing the validity data for the top level of a list column. It ends up writing out an all-valid mask, when there are actually nulls. Confirmed this is a writer issue by loading the resulting file in external tools as well. ORC writer does not exhibit this behavior.

Repro:

constexpr int num_rows = 16;

std::mt19937 gen(6542);
std::bernoulli_distribution bn(0.7f);
auto valids =
  cudf::detail::make_counting_transform_iterator(0, [&](int index) { return bn(gen); });
auto values = thrust::make_counting_iterator(0);

// list<float>
constexpr int floats_per_row = 4;
auto c1_offset_iter = cudf::detail::make_counting_transform_iterator(0, [floats_per_row](cudf::size_type idx){
    return idx * floats_per_row;
  });
cudf::test::fixed_width_column_wrapper<cudf::offset_type> c1_offsets(c1_offset_iter, c1_offset_iter + num_rows + 1);    
cudf::test::fixed_width_column_wrapper<float> c1_floats(values, values + (num_rows * floats_per_row), valids);
auto c1 = cudf::make_lists_column(num_rows,
                                  c1_offsets.release(),
                                  c1_floats.release(),
                                  cudf::UNKNOWN_NULL_COUNT,
                                  cudf::test::detail::make_null_mask(valids, valids + num_rows));

cudf::test::print(*c1);

// write it out
cudf::table_view tbl({*c1});  
std::string filepath("bad_list_write.parquet");
cudf::io::parquet_writer_options out_args =
  cudf::io::parquet_writer_options::builder(cudf::io::sink_info{filepath}, tbl);
cudf::io::write_parquet(out_args);

cudf::io::parquet_reader_options read_args =
  cudf::io::parquet_reader_options::builder(cudf::io::source_info{filepath});
auto result = cudf::io::read_parquet(read_args);

cudf::test::print(result.tbl->get_column(0));

Expected (nested float column not showed)

List<float>:
Length : 16
Offsets : 0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64
Null count: 5
1111111010011010

Got

List<float>:
Length : 16
Offsets : 0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64
Null count: 0
1111111111111111

[vuule]

The issue only reproes with list columns that have non-empty null elements. Modifying the failing test to have zero-length null lists fixes the test.
Such input validation is (arguably) out of scope of the Parquet writer. cuDF APIs assume that list columns are sanitized.

[ttnghia]

So we may end up to always calling purge_nonempty_nulls before writing (of course after checking has_nulls)?

[vuule]

It depends on where the data is coming from. For example, cuIO readers do not create such columns.

Maybe @mythrocks can comment on the cases in which we can create non-conforming columns.

[GregoryKimball]

I will convert this into a documentation issue. The update will be needed in https://github.com/rapidsai/cudf/blob/branch-22.12/cpp/doxygen/developer_guide/DEVELOPER_GUIDE.md#list-columns

[bdice]

I think this is solved by #11853 unless we want to copy the advice into another location.

https://github.com/rapidsai/cudf/blob/branch-22.12/cpp/doxygen/developer_guide/DEVELOPER_GUIDE.md#libcudf-expects-nested-types-to-have-sanitized-null-masks

[vyasr]

Agreed. Closing.