[BUG] listReduce on a single row input with an empty list generates a zero row output

[gerashegalov]

Describe the bug
While testing ArrayExists in spark-rapids we hit an exception when a partition consists of a single row with an empty list. The root cause is somewhere behind the fact that listReduce produces an empty output instead of maintaining the invariant: output row count equals input row count

Steps/Code to reproduce bug

The repro via JVM in Scala REPL is as follows.

val dt = new HostColumnVector.ListType(true, new HostColumnVector.BasicType(true, DType.BOOL8))
val singleEmptyListCV = ColumnVector.fromLists(dt, java.util.Arrays.asList())
// singleEmptyListCV: ai.rapids.cudf.ColumnVector = ColumnVector{rows=1, type=LIST, nullCount=Optional.empty, offHeap=(ID: 5 7fb64ad20650)}
val singleEmptyListReducedCV = singleEmptyListCV.listReduce(SegmentedReductionAggregation.any(), NullPolicy.INCLUDE, DType.BOOL8)
// singleEmptyListReducedCV: ai.rapids.cudf.ColumnVector = ColumnVector{rows=0, type=BOOL8, nullCount=Optional.empty, offHeap=(ID: 6 7fb5b2469d20)}
val gpuCVClz = ShimLoader.loadClass("com.nvidia.spark.rapids.GpuColumnVector")
MethodUtils.invokeStaticMethod(gpuCVClz, "debug", "bad", singleEmptyListReducedCV)
// GPU COLUMN bad - NC: 0 DATA: null VAL: null
// COLUMN bad - BOOL8

The output is as expected if the input includes another row with a non-empty list

val emptyAndNonEmptyListCV = ColumnVector.fromLists(dt, java.util.Arrays.asList(), java.util.Arrays.asList(false))
// emptyAndNonEmptyListCV: ai.rapids.cudf.ColumnVector = ColumnVector{rows=2, type=LIST, nullCount=Optional.empty, offHeap=(ID: 13 7fb5a8171ce0)}
val emptyAndNonEmptyListReducedCV = emptyAndNonEmptyListCV.listReduce(SegmentedReductionAggregation.any(), NullPolicy.INCLUDE, DType.BOOL8)
// emptyAndNonEmptyListReducedCV: ai.rapids.cudf.ColumnVector = ColumnVector{rows=2, type=BOOL8, nullCount=Optional.empty, offHeap=(ID: 14 7fb5b6de1020)}
MethodUtils.invokeStaticMethod(gpuCVClz, "debug", "good", emptyAndNonEmptyListReducedCV)
// GPU COLUMN good - NC: 1 DATA: DeviceMemoryBufferView{address=0x7fb5bfc01000, length=2, id=-1} VAL: DeviceMemoryBufferView{address=0x7fb5bfc01400, length=64, id=-1}
// COLUMN good - BOOL8
// 0 NULL
// 1 false

Expected behavior
Reduction of a single row with an empty list should produce a single row output

Environment overview (please complete the following information)

• Environment location: any
• Method of cuDF install: from source

Environment details
TBD

Additional context
NVIDIA/spark-rapids#5108

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[jlowe]

This appears to be a bug in libcudf rather than the cudf Java/JNI layer.

[jlowe]

Apply the following patch and build gtests/REDUCTION_TEST to reproduce the issue in C++:

diff --git a/cpp/tests/reductions/segmented_reduction_tests.cpp b/cpp/tests/reductions/segmented_reduction_tests.cpp
index 8a9a8fb549..2f3044ea4f 100644
--- a/cpp/tests/reductions/segmented_reduction_tests.cpp
+++ b/cpp/tests/reductions/segmented_reduction_tests.cpp
@@ -107,6 +107,27 @@ TYPED_TEST(SegmentedReductionTest, MaxExcludeNulls)
   CUDF_TEST_EXPECT_COLUMNS_EQUAL(*res, expect);
 }
 
+TYPED_TEST(SegmentedReductionTest, MaxExcludeNullsAllNull)
+{
+  // [null]
+  // values:    {}
+  // offsets:   {0, 0}
+  // nullmask:  {1}
+  // outputs:   {XXX}
+  // output nullmask: {1}
+  auto input     = fixed_width_column_wrapper<TypeParam>{};
+  auto offsets   = std::vector<size_type>{0,0};
+  auto d_offsets = thrust::device_vector<size_type>(offsets);
+  auto expect = fixed_width_column_wrapper<TypeParam>{{XXX}, {1}};
+
+  auto res = segmented_reduce(input,
+                              d_offsets,
+                              *make_max_aggregation<segmented_reduce_aggregation>(),
+                              data_type{type_to_id<TypeParam>()},
+                              null_policy::EXCLUDE);
+  CUDF_TEST_EXPECT_COLUMNS_EQUAL(*res, expect);
+}
+
 TYPED_TEST(SegmentedReductionTest, MinExcludeNulls)
 {
   // [1, 2, 3], [1, null, 3], [1], [null], [null, null], []

The problem occurs when the lists column is all nulls and the child column is empty (because there is no need for backing values in a list of all nulls).

[davidwendt]

I just want to verify that the expected result is a single row column (size()==1) where row 0 is null.
So the above test case should actually have the expect column initialized as:

auto expect = fixed_width_column_wrapper<TypeParam>{{XXX}, {0}};

@jlowe

[jlowe]

the expected result is a single row column (size()==1) where row 0 is null.

Yes, apologies for the bug in the test.