Fix logical type issues in the Parquet writer #14322
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vuule
added
bug
etseidl
reviewed
Oct 24, 2023
3 tasks
|
CC @galipremsagar @rjzamora for impact on the Python side |
…bug-write_parquet-adjusted-to-utc
jlowe
reviewed
Oct 25, 2023
…bug-write_parquet-adjusted-to-utc
…m/vuule/cudf into bug-write_parquet-adjusted-to-utc
vuule
changed the title
|
Thank you @etseidl for the |
There was a problem hiding this comment.
@vuule Can address the todo here(https://github.com/rapidsai/cudf/pull/14327/files#diff-ba3ea8b1c9c17ff8301a86d6a844b075f4f49feaa782f4a8f190e631123b007eR484) since this PR is fixing #14326
davidwendt
approved these changes
Oct 27, 2023
…bug-write_parquet-adjusted-to-utc
…m/vuule/cudf into bug-write_parquet-adjusted-to-utc
hyperbolic2346
approved these changes
Oct 30, 2023
|
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Closes #14315
Closes #14326
Parquet writer writes time and timestamp types with logical type with
isAdjustedToUTCasfalse. However, timestamps in libcudf tables are implicitly in UTC and don't need to be adjusted.This PR changes the
isAdjustedToUTCto true.Also added a writer option to write timestamps as local, as this is the expected behavior on the Python side.
Also changed the way logical type is handled for UNKNOWN type columns in
merge_row_group_metadata- the logical type is excluded from merged metadata because of issues with type inference.Checklist