Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(python): don't convert "ns"-precision temporal types via pyarrow #6592

Merged
merged 1 commit into from
Jan 31, 2023
Merged

fix(python): don't convert "ns"-precision temporal types via pyarrow #6592

merged 1 commit into from
Jan 31, 2023

Conversation

alexander-beedie
Copy link
Collaborator

Closes #6591.

Temporal values loaded via to_dicts or iter_rows in columns with "ns" precision shouldn't go via pyarrow, as it converts them to pandas Timestamp/Timedelta types.

@github-actions github-actions bot added fix Bug fix python Related to Python Polars labels Jan 31, 2023
ritchie46
ritchie46 reviewed Jan 31, 2023
View reviewed changes
Copy link
Member

@ritchie46 ritchie46 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only the case with ns precision?

py-polars/polars/internals/dataframe/frame.py Outdated Show resolved Hide resolved
@alexander-beedie
Copy link
Collaborator Author

alexander-beedie commented Jan 31, 2023 •
edited
Loading

Is this only the case with ns precision?

Yes - everything less than that has a python-native representation; it's only ns where it loads via pandas Timestamp (or pandas Timedelta in the case of ns duration).

(Added a Notes section to the relevant functions about potential loss of ns precision, and made it clearer that we're returning python-native values here).

@ritchie46 ritchie46 merged commit c2bfb37 into pola-rs:master Jan 31, 2023
@alexander-beedie alexander-beedie deleted the dont-create-pandas-temporal-types branch February 2, 2023 08:39
Vincenthays pushed a commit to Vincenthays/polars that referenced this pull request Feb 9, 2023
@alexander-beedie
fix(python): don't convert "ns"-precision temporal types via pyarrow (
729f25a 
pola-rs#6592)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ritchie46 ritchie46 ritchie46 left review comments

Assignees
No one assigned
Labels
fix Bug fix python Related to Python Polars
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

polars 0.16.1: using to_dicts datetime with timezone change type to pandas Timestamp instead of datetime
2 participants
@alexander-beedie @ritchie46