Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Support reading Int as Byte/Short/Date from parquet #5445

Closed
sperlingxx opened this issue May 10, 2022 · 0 comments
Closed

[FEA] Support reading Int as Byte/Short/Date from parquet #5445

sperlingxx opened this issue May 10, 2022 · 0 comments
Assignees
@sperlingxx
Labels
feature request New feature or request

Comments

@sperlingxx
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Spark supports reading parquet INT32 column as a byte/short/date column. We need to add corresponding downcast converters into GpuParquet readers.
@sperlingxx sperlingxx added feature request New feature or request ? - Needs Triage Need team to review and classify labels May 10, 2022
@sperlingxx sperlingxx self-assigned this May 10, 2022
@sperlingxx sperlingxx mentioned this issue May 10, 2022
Merged
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label May 10, 2022
@revans2 revans2 mentioned this issue May 11, 2022
Closed
sperlingxx added a commit that referenced this issue May 12, 2022
@sperlingxx
Check schema compatibility when building parquet readers (#5434)
e41a6f3 
Fixes #5200 #5445

This PR is to add the schema check, referring the checking process of Spark. Converters downcasting INT32 to Byte/Short/Date are added in this PR as well. Note: This PR uses some deprecated API of parquet-mr, in order to accommodate Spark 3.1.

Signed-off-by: sperlingxx <lovedreamf@gmail.com>
@sameerz sameerz mentioned this issue May 12, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sperlingxx sperlingxx

Labels
feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mattahrens @sperlingxx