Parquet coalesce file reader for local filesystems

[tgravescs]

Add back in support for the parquet coalesce file reader. We introduced this in 0.2 here dcd119c#diff-6c8f47c497382f147106e8e82020d1828eee055c2f2a7c6d60807262e8b63cf9
It was then replaced with the multi-threaded parquet reader which worked much better in the cloud. Unfortunately I didn't do enough testing on local file systems (file: and hdfs:). The coalesce reader works much better there. So I'm adding it back and we have 3 parquet readers now. The original per file reader, the multi-threaded reader which works well with cloud, then the coalesce reader which works well with local filesystems.

This implementation is very similar to the one we had before (code was just copied) with the addition of it now does the copy of blocks from a single file to the host memory buffer in background threads. We allocate the buffer, slice it, then run the actual copy in parallel in separate threads and wait for it to be done, then pass that single buffer down to gpu.

Note this required me to put back the checks for the inputFileName and blocks and if we find the user using those we change it to not use the coalescing reader.

There is now one config that you select the reader you want to use, by default its on auto where it chooses either the coalescing of the multi-threaded based not he cloud filesystem scheme.

I hardcoded some filesystem schemes as to what I knew were cloud based filesystems then have a config so user can add more: spark.rapids.cloudSchemes. I thought about putting all the schemes in there but figure you can turn off one of the other 2 configs if you really want it to use one of the other.

[tgravescs]

build

[tgravescs]

build

[tgravescs]

build

[tgravescs]

build

[chenrui17]

It was then replaced with the multi-threaded parquet reader which worked much better in the cloud. Unfortunately I didn't do enough testing on local file systems (file: and hdfs:).

@tgravescs

if i use HDFS , it woud be go to the local file systems ? but in another #1200 , you said i should set the spark.rapids.sql.format.parquet.reader.type="MULTITHREADED" , i am confused . please help me .

btw , I use HDFS , how to confirme my multi-thread read small parquet is working well . my parquet file is about 250MB , and executor.cores = 16 . I attemp to set spark.rapids.sql.format.parquet.multiThreadedRead.numThreads = 48 , but query execution time is almost same .

[tgravescs]

Please do not ask questions on closed PRs, an issue would be much better. In testing the coalesce reader works better for local file system (ie local disks on same node as the executors). But that is assuming the disks are fast enough and I think the case you ran into is partitioning. So it might not always be the case, which is why it's configurable.