Add option to control number of partitions when converting from CSV to Parquet

[andygrove]

Signed-off-by: Andy Grove andygrove@nvidia.com

When we convert TPC-* CSV files to Parquet, the number and size of Parquet files created are not consistent and depends on the number of executor cores running the conversion. This can lead to thousands of small files being created, which is not optimal for GPU.

This PR adds the option to control the number of partitions per table using coalesce or repartition.

Also, all three TPC-* benchmarks now have a ConvertFiles object with a main method, so that the file conversion can be submitted with spark-submit with command-line arguments for all available options. For TPC-DS, there is an option to use partitioning when creating the Parquet files since the underlying code supports that.

This closes #902

[tgravescs]

any reason to not support also growing the number of partitions? Basically just have this numPartitions. You would need to use repartition in that case. I would find this useful for like small file testing where I actually want to grow the number of partitions

[andygrove]

Good point. I can look at this later today.

[andygrove]

@tgravescs I'm looking at this now and there are multiple options for doing this that I can see.

Option 1: Add specific command-line arguments for coalesce vs repartition. For the issue where some tables have too many files, coalesce is much more efficient than repartition, and for the tables that already have a small number of files, I don't want to increase the number.

Option 2: Dynamically look at the number of partitions in each table and then decide whether to coalesce or repartition to achieve the desired number of partitions, but this would mean that we could end up increasing partitions for some tables and decreasing for others.

I think option 1 is more explicit and would work well for both of our use cases? I'll start down this path but let me know what you think.

[jlowe]

I've used coalesce in the past and it works great, but it can be problematic when applying it blindly across all tables in the dataset. For example, there are typically many tables in these benchmarks that scale very slowly or not at all (e.g.: tables containing regions, warehouse locations, etc.) that for the best performance should just be one file. But then there are other tables that have the largest scale (e.g.: tables containing individual sales record data) that would be disastrous to force into a single file.

This may need to be handled in a benchmark-specific way, e.g.: tiny tables that don't really scale with the rest of the data are always coalesced into a single table, and there are two user-specified settings, one for the number of partitions to use for "medium" scale tables, and another for "large" scale tables.

[andygrove]

Perhaps it would make sense to allow the --coalesce and --repartition arguments to operate on individual tables as a starting point e.g. --coalesce table1=1 table2=24 --repartition table3=200 so that we have the ability to fully control partition sizes manually and then we can build higher-level logic on top in the future to do this in a more automated way.

[andygrove]

These changes have been tested with TPC-DS but not with TPC-H or TPC-xBB.

[andygrove]

build

[tgravescs]

so if I specify it in both I get the coalesce? Perhaps we should error?

[andygrove]

I'm not sure what happened, but after applying the suggested changes, I see other changes in this PR. I will rebase.

[andygrove]

@jlowe @tgravescs Thanks for the reviews. I've addressed your feedback now.

[andygrove]

build

[andygrove]

build

[andygrove]

build

[jlowe]

build