-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to read hive-style partitioned parquet file using read_parquet
#10276
Comments
There is import pyarrow.dataset as ds
pl.scan_pyarrow_dataset(ds.dataset("my_table_name")).collect()
# shape: (13, 2)
# ┌─────────┬─────────┐
# │ USER_ID ┆ TRX_CNT │
# │ --- ┆ --- │
# │ f64 ┆ f64 │
# ╞═════════╪═════════╡
# │ 1000.0 ┆ 434.0 │
# │ 1001.0 ┆ 11.0 │
# │ 1002.0 ┆ 3.0 │
# │ 1003.0 ┆ 555.0 │
# │ … ┆ … │
# │ 1001.0 ┆ 21.0 │
# │ 1003.0 ┆ 44.0 │
# │ 1005.0 ┆ 111.0 │
# │ 1008.0 ┆ 222.0 │
# └─────────┴─────────┘ |
This is correct if all the files are in the same directory, but otherwise (as @cmdlineluser says) you need to use (I've just committed a small update to the docs that adds a more explicit note to |
I think Not sure how hard this is to implement, but it should be a goal, in my opinion. |
Could pl.read_parquet() just call this pl.scan_pyarrow_dataset() function? |
read_parquet
(Updated the title to clarify the issue more specifically, so we can reference it easily later). |
Added my last comment as a separate issue: [https://github.com//issues/10296] |
|
This should be fixed by #13044 If not, please comment and I can reopen this issue. |
Hi,
In polars 0.20.5 this works:
pl.read_parquet('my_table_name/**/*.parquet')
but this:
pl.read_parquet('my_table_name')
throws the bellow error:
IsADirectoryError: expected a file path; 'my_table_name' is a directory
Can I not use a directory here? (I should be able to)
Thanks,
Lmocsi
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Vírusmentes.www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
…On Fri, Jan 12, 2024 at 11:43 PM Stijn de Gooijer ***@***.***> wrote:
This should be fixed by #13044
<#13044>
If not, please comment and I can reopen this issue.
—
Reply to this email directly, view it on GitHub
<#10276 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APOYDWZQGT2KDV76BDFLRDLYOG4ATAVCNFSM6AAAAAA3DB4GP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJQGA3TONZUGM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of Polars.
Reproducible example
"""
I have a partitioned parquet file like this (filename is
my_table_name, and it is partitioned by calendar_date column, see attached):
"""
"""
It is giving me the error:
ComputeError: error while reading /path/my_table_name/CALENDAR_DATE=2023-08-01 00%3A00%3A00: External format error: File out of specification: underlying IO error: Invalid argument (os error 22)
"""
my_table_name.zip
Issue description
It seems, that as of now polars is supporting only two types of parquet structure:
Though, true partitioned parquet files (where you have separate directory for each partition) does not seem to be supported. :(
Expected behavior
Able to read the partitioned parquet file, even as a lazy dataframe (just like Spark does)
Installed versions
The text was updated successfully, but these errors were encountered: