Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test compatibility between pyarrow and GPU #9288

Closed
res-life opened this issue Sep 22, 2023 · 1 comment
Closed

Test compatibility between pyarrow and GPU #9288

res-life opened this issue Sep 22, 2023 · 1 comment
Labels
test Only impacts tests

Comments

@res-life
Copy link
Collaborator

res-life commented Sep 22, 2023 •
edited
Loading

This is a sub-issue from #8666
Test compatibility between pyarrow and GPU.

  • write/read by pyarrow and write/read by GPU
  • try to generate files in parquet testing cases on pyarrow to test corner case. Refer to related test cases
    It's hard to for pyarrow API to generate these kind of files via current pyarrow API, so remove this item. And add the following item.
  • Try to use different parameters for pyarrow API to generate parquet file, then compare compatibility.

CPU encodings: BIT_PACKED,RLE,PLAIN_DICTIONARY.
GPU encodings: RLE,PLAIN,PLAIN_DICTIONARY.
The GPU reading pyarrow generated file may fail because of different encodings.
@res-life res-life mentioned this issue Sep 22, 2023
Open
47 tasks
@res-life res-life changed the title Written/read by pyarrow and read/written by GPU. Generate files in parquet testing cases. CPU encodings: BIT_PACKED,RLE,PLAIN_DICTIONARY. GPU encodings: RLE,PLAIN,PLAIN_DICTIONARY. The GPU reading pyarrow generated file may fail because of different encodings. Written/read by pyarrow and read/written by GPU. Sep 22, 2023
@res-life res-life changed the title Written/read by pyarrow and read/written by GPU. write/read by pyarrow and write/read by GPU. Sep 22, 2023
@res-life res-life mentioned this issue Sep 22, 2023
Merged
@res-life res-life changed the title write/read by pyarrow and write/read by GPU. Test compatibility between pyarrow and GPU Sep 28, 2023
@thirtiseven thirtiseven mentioned this issue Oct 9, 2023
Merged
@sameerz sameerz added the test Only impacts tests label Oct 24, 2023
@res-life
Copy link
Collaborator Author

Close it via #9289

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
test Only impacts tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sameerz @res-life