feat: support 'hive partitioning' aware readers

[ritchie46]

closes #10980, #10276

[alexander-beedie]

Nice! Hive partitioning support is super-useful 👍

[ritchie46]

Nice! Hive partitioning support is super-useful 👍

Yes, especially the savings are potentially huge!

[ion-elgreco]

Closes also this: #10276

[uditrana]

I am noticing an interesting pattern/bug with this feature.
I wrote a large dataset out using

df.write_parquet(
            file=DIR,
            use_pyarrow=True,
            pyarrow_options={
                "partition_cols": ["part_id_1", "part_id_2"],
                "basename_template": "test_{i}.parquet",
                "existing_data_behavior": "overwrite_or_ignore",
            },
        )

When I read these files back in using pl.read_parquet(), I observe 2 different results:

df1 = pl.read_parquet(DIR / "part_id_1=1" / "part_id_2="A" / "test_0.parquet")
df2 = pl.read_parquet(DIR / "part_id_1=1" / "part_id_2="A" / "*test_0.parquet")

In df1, part_id_1 and part_id_2 are missing from the columns, while they are present in df2. Basically, seems that read_parquet will not look for hive-partitioning structure if it is passed a single file, and even though in the second case it is reading only one file... the fact that the pattern could include multiple causes it to pick up the structure correctly.

[uditrana]

Also, slightly tangentially, could it make sense to expose an option in pl.read_parquet to include an option to turn off schema extension for hive partition columns? I have a use case where I am running into this error while trying to read a dataset:

ComputeError: invalid hive partitions

Extending the schema with the hive partitioned columns creates duplicate fields.

This dataset was written in a way to keep the partition columns in the partitioned dataframes so that other libraries (like Pandas) can read the dataframes individually and concatenate serially (since they don't support Hive natively yet).