Add ignoreCorruptFiles for ORC readers [databricks]

[wbo4958]

This PR is to close #4802, #4803 and #4795.

Since ORC/Parquet PERFILE and texted-based GPU readings (CSV, JSON) are already in the scope of FileScanRDD which has done the feature ignoring missing and ignoring corrupt files. we don't need to do extra things for this.

I can't find a way to trigger the IOException or RuntimeException for CPU CSV/JSON parsing, since CPU can parse wrong files when CSV/JSON reading.

But for JSON corrupt files, CUDF will throw an exception, I just add a test for GPU JSON reading.

[wbo4958]

build

[jlowe]

Doesn't GpuTextBasedPartitionReader need a try..catch block around readToTable(Boolean) to cover the case of the filesystem returning an I/O exception or runtime exception? Spark CPU covers this in a central place via FilePartitionReader.next which skips files that return I/O errors or other runtime exceptions when ignoreCorruptFiles is true.

[wbo4958]

Doesn't GpuTextBasedPartitionReader need a try..catch block around readToTable(Boolean) to cover the case of the filesystem returning an I/O exception or runtime exception? Spark CPU covers this in a central place via FilePartitionReader.next which skips files that return I/O errors or other runtime exceptions when ignoreCorruptFiles is true.

Yeah, we don't need to do that, since Spark itself has helped us for this.

For V1, we will get FileScanRDD or GpuFileScanRDD, both of which has implemented ignoreCorruptFiles/ignoreMissingValues

For V2, we will get GpuDataSourceRDD, and the GpuTextBasedPartitionReader will be wrapped into FilePartitionReader which has implemented the ignoreCorruptFiles/ignoreMissingValues.

I added the test_json_read_with_corrupt_files for both v1 and v2 scenarios, which can tested the above scenarios.