Add support for Delta Lake 2.4.0 [databricks]

[andygrove]

Part of #8547

Closes #8556

This PR adds shims for Delta Lake 2.4.0, which is only compatible with Spark 3.4.x

The approach follows the current approach for Delta Lake shims, where each version is largely the same code, with minor differences. This makes it easier to diff against the Delta Lake project.

I started by copying the 22x shims as the basis for 24x, then performed a diff with the Delta Lake 2.4 source and pulled in some changes.

The scope of this PR is to get current Delta Lake tests passing with 2.4, and does not aim to support all new features in Delta Lake 2.4 on the GPU.

Delta Lake 2.4.0 release notes: https://github.com/delta-io/delta/releases/tag/v2.4.0

Follow-on issues:

• [FEA] [Delta Lake] Add support for deletion vectors in OSS Delta Lake #8554
• [FEA] [Databricks 12.2] Support GpuMergeIntoCommand notMatchedBySourceClauses on GPU #8415

[andygrove]

build

[andygrove]

Moving this back to draft while I audit differences to Delta Lake implementation

[gerashegalov]

for easier maintenance in the future, I would create a separate variable with the list of metrics one per line,

metrics_to_remove = [
  'executionTimeMs', 
  'numOutputBytes', 
  ...
]

nit: I'd stick to some consistent use of quotes or double-quotes in a single code fragment at least

[andygrove]

build

[andygrove]

build

[andygrove]

build

[andygrove]

build

[andygrove]

build

[andygrove]

build

[andygrove]

Integration tests are failing on Databricks 12.2 due to GC pauses causing timeouts. I can reproduce manually by running delta_lake_merge_test.py. Here are examples of warnings that I see:

23/07/07 18:18:33 WARN DBRDebuggerEventReporter: Driver/10.59.237.142 paused the JVM process 110 seconds during the past 120 seconds (92.09%) because of GC. We observed 105 such issue(s) since 2023-07-07T18:13:26.473Z.

23/07/07 18:20:56 WARN TaskMemoryManager: Failed to allocate a page (8388608 bytes), try again.

[jlowe]

We had problems in the past due to aggressive caching of Delta Log snapshots. Wonder if the same thing or something similar is happening. We should be specifying -Ddelta.log.cacheSize=10 during the tests, make sure that's getting through. Could get a heap dump, maybe there's another cache that needs a limit?

[andygrove]

It looks like I went too far when refactoring setup_dest_tables. I have reverted some of those changes and verified that delta_lake_merge_test and delta_lake_delete_test now run without issue on DBR 12.2

[andygrove]

build