Fallback to CPU for Parquet reads with _databricks_internal columns [databricks]

[andygrove]

When performing a Delta Lake MERGE operation on Databricks, a query runs that relies on receiving valid data from computed columns such as _databricks_internal_edge_computed_column_row_index and _databricks_internal_edge_computed_column_skip_row in the Parquet read.

For example:

+- FileScan parquet [country#755,_databricks_internal_edge_computed_column_row_index#963L] Batched: true, DataFilters: [], Format: Parquet, Location: TahoeBatchFileIndex(1 paths)[file:/tmp/myth/delta/corpus], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<country:string,_databricks_internal_edge_computed_column_row_index:bigint>

These columns obviously do not exist if we are reading the files on GPU so we need to fall back to CPU in this case.

This PR is still WIP because there are no tests yet.

[andygrove]

I will look at adding a shim so we only apply this check on Databricks

[mythrocks]

👍 I have verified that this change allows for the MERGE command to fall back to CPU mode.
(Note that I tested this change alongside #6230.)

I'm inclined to call this "Plan A", instead of it being the fallback position for 22.08. The _databricks_internal_ columns might have implementation-specific connotations (for computing cardinality, etc.). It might be best not to attempt GPU acceleration for it right now.

We have verified that run-of-the-mill DELTA table reads are not affected: only the MERGE query is falling back.

[andygrove]

build

[andygrove]

build

[andygrove]

I still plan on making the shim change, but tests are now passing, and this could be merged as is for 22.08 with a follow-up issue to use shims in 22,10.

[andygrove]

I filed #6279 for the follow-up issue