Add support for writing deletion vectors in Delta Lake 2.4

[andygrove]

Closes #8554

Changes in this PR:

• Stop falling back to CPU in GpuDeleteCommand when deletion vectors are enabled
• Implement logic from OSS Delta Lake 2.4 for writing deletion vectors. Note that we are not necessarily GPU-accelerating anything here but just matching the behavior of OSS Delta Lake
• Add test to ensure behavior is correct
• Update Delta Log comparison to handle deletionVector structures

Note that GPU-accelerating the metadata queries involved will not be trivial due to row-based UDFs, custom data types, and roaring bitmap aggregation operators, which we do not support on GPU.

[andygrove]

test

[andygrove]

build

[andygrove]

Databricks build failing due to #8726

[andygrove]

build

[andygrove]

build

[andygrove]

build

[andygrove]

build

[andygrove]

build

[andygrove]

build failed with seemingly unrelated issue:

FileCacheIntegrationSuite:^[[0m
- filecache metrics v1 Parquet^[[0m
- filecache metrics v2 Parquet^[[0m
- filecache metrics v1 ORC *** FAILED ***^[[0m
  0 was not greater than 0 (FileCacheIntegrationSuite.scala:175)

[andygrove]

build

[jlowe]

It doesn't feel like we're properly supporting deletion vectors if we're sending the bulk of the data through a CPU UDF. That appears to be the case after glancing at what DeleteWithDeletionVectorsHelper.findTouchedFiles does.

Also what about spark332db which also supports deletion vectors?

[jlowe]

Note that we've built GPU versions of Delta Lake CPU UDFs before, and we should be able to do something similar here if necessary, although I haven't scoped the effort.

If we've done benchmarking showing that partially supporting deletion vectors, (i.e.: falling back to the CPU to compute the vector values before writing via GPU), significantly outperforms falling back to the CPU to do the entire delete operation, then this would be worth committing.

[andygrove]

Also what about spark332db which also supports deletion vectors?

I'm working on that as a separate PR since it is much more involved. The tracking issue is #8654

[andygrove]

Note that we've built GPU versions of Delta Lake CPU UDFs before, and we should be able to do something similar here if necessary, although I haven't scoped the effort.

Af first glance, it does not look trivial to implement on the GPU, since it looks like we would need to support roaring bitmap format vectors.

If we've done benchmarking showing that partially supporting deletion vectors, (i.e.: falling back to the CPU to compute the vector values before writing via GPU), significantly outperforms falling back to the CPU to do the entire delete operation, then this would be worth committing.

We have not benchmarked this. I will move this PR to draft for now but perhaps this should just be closed until we decide to fully GPU-accelerate the deletion vector writes.