Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Support struct types in ORC writer #7830

Closed
1 of 2 tasks
vuule opened this issue Apr 2, 2021 · 2 comments · Fixed by #9025
Closed
1 of 2 tasks

[FEA] Support struct types in ORC writer #7830

vuule opened this issue Apr 2, 2021 · 2 comments · Fixed by #9025
Assignees
@vuule
Labels
cuIO cuIO issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API.
Milestone
IO Data Type Expa...

Comments

@vuule
Copy link
Contributor

vuule commented Apr 2, 2021 •
edited by beckernick
Loading

Struct support:
@vuule vuule added feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API. cuIO cuIO issue labels Apr 2, 2021
@sameerz sameerz mentioned this issue Apr 20, 2021
Closed
@github-actions
Copy link

github-actions bot commented May 2, 2021

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

@vuule vuule mentioned this issue Jun 4, 2021
Closed
@beckernick
Copy link
Member

beckernick commented Jul 19, 2021 •
edited
Loading

As struct support in ORC reader was merged in #8599 and not linked to this issue, I'm going to update this issue to explicitly be about struct support in the writer

@beckernick beckernick changed the title [FEA] Support struct types in ORC reader and writer [FEA] Support struct types in ORC writer Jul 19, 2021
@vuule vuule self-assigned this Jul 22, 2021
@beckernick beckernick mentioned this issue Jul 26, 2021
Closed
@vuule vuule mentioned this issue Jul 27, 2021
Closed
@vuule vuule mentioned this issue Sep 14, 2021
Merged
rapids-bot bot pushed a commit that referenced this issue Sep 22, 2021
@vuule
Add support for struct type in ORC writer (#9025)
2c6b39b 
Fixes #7830, #8443

Features:
- Use the new table metadata type that matches the table hierarchy, `table_input_metadata`.
- Support struct columns in the writer.

Changes:
- Null masks are encoded as aligned rowgroups to avoid invalid bits when the number of encoded rows is not divisible by 8 (except for the last rowgroup in each stripe). This also affects list columns. The issue is equivalent to #6763 (boolean columns only).
- Added pushdown masks that are used to determine which child elements should not be encoded, including null mask bits.
- Use pushdown masks for rowgroup alignment, null mask encoding and value encoding.
- Separated the null mask encoding from value encoding - can be further moved to a separate kernel call.

Breaking because the table metadata type has changed.

Authors:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Jason Lowe (https://github.com/jlowe)

Approvers:
  - Robert Maynard (https://github.com/robertmaynard)
  - AJ Schmidt (https://github.com/ajschmidt8)
  - Robert (Bobby) Evans (https://github.com/revans2)
  - Vyas Ramasubramani (https://github.com/vyasr)
  - Devavret Makkar (https://github.com/devavret)
  - Ram (Ramakrishna Prabhu) (https://github.com/rgsl888prabhu)

URL: #9025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vuule vuule

Labels
cuIO cuIO issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API.
Projects
None yet
Milestone
IO Data Type Expansion
Development

Successfully merging a pull request may close this issue.

Add support for struct type in ORC writer vuule/cudf
3 participants
@randerzander @beckernick @vuule