[FEA] Unify cudf::structs::detail::flatten_nested_columns
and cudf::experimental::decompose_structs
to improve performance for structs comparison
#13032
Labels
0 - Backlog
In queue waiting for assignment
libcudf
Affects libcudf (C++/CUDA) code.
Performance
Performance related issue
proposal
Change current process or code
Milestone
For comparing structs column, both the legacy row comparators and the new experimental row comparators rely on struct flattening procedures. Each of them have their own flattening mechanism:
cudf::structs::detail::flatten_nested_columns
andcudf::experimental::decompose_structs
. The difference between them are:cudf::structs::detail::flatten_nested_columns
replaces the input structs column with an optional column generated by materializing the input null mask.cudf::experimental::decompose_structs
doesn't materialize any new column. Instead, it replaces the input structs column with a modified version of it, which only has either zero or one child at the innermost level.Although these APIs produce different output, these APIs do very similar job:
The issue of each from these approaches are:
cudf::structs::detail::flatten_nested_columns
needs to materialize null mask of the input column into a real column.cudf::experimental::decompose_structs
still has a nested structs column in the output. Although that column only has zero or one child at the innermost level, it still causes performance degradation if its nested level is very high.As such, we can unify the two approaches, taking the pros of both while eliminating the cons. The new flattening API should:
This seems to be very straightforward with modifying the existing
cudf::experimental::decompose_structs
API.The text was updated successfully, but these errors were encountered: