Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add better automated tests for Arrow columnar copy in HostColumnarToGpu #1665

Merged
merged 9 commits into from
Feb 4, 2021
Merged

Add better automated tests for Arrow columnar copy in HostColumnarToGpu #1665

merged 9 commits into from
Feb 4, 2021

Conversation

tgravescs
Copy link
Collaborator

  1. moved the test to end in test.py so that pytest regex with -k picked itup
  2. Found a bug in the HostcolumnarToGPU without Arrow where it didn't handle Boolean properly
  3. changed to have some internal Test classes for DataSourceV2 so we didn't have to pull external jar. Add test for all the supported types currently. Much of this code was copied from Spark data source v2 tests or ArrowUtils and then modified to my needs.

fixes #1620

@tgravescs tgravescs added the test Only impacts tests label Feb 3, 2021
@tgravescs tgravescs added this to the Feb 1 - Feb 12 milestone Feb 3, 2021
@tgravescs tgravescs self-assigned this Feb 3, 2021
@tgravescs
Copy link
Collaborator Author

build

jlowe
jlowe previously approved these changes Feb 3, 2021
View reviewed changes
@jlowe
Copy link
Member

jlowe commented Feb 3, 2021

Looks like a scalastyle line length violation needs to be fixed.

@tgravescs
fix line length
37c66f9 
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
@tgravescs
Copy link
Collaborator Author

build

@tgravescs tgravescs merged commit 88a6a58 into NVIDIA:branch-0.4 Feb 4, 2021
@tgravescs tgravescs deleted the arrowTesting branch February 4, 2021 14:01
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@tgravescs
Add better automated tests for Arrow columnar copy in HostColumnarToG…
38353d7 
…pu (NVIDIA#1665)

* Add Data source v2 test classes

Signed-off-by: Thomas Graves <tgraves@nvidia.com>

* update v2 source testing

* fix batch num rows and logging

* update the numberin batch

* Fix issue with reading booleans from ArrowColumnVectors and add more
tests

* move test file so pytest regex pick it up

* add comments

* fix line length

Signed-off-by: Thomas Graves <tgraves@nvidia.com>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@tgravescs
Add better automated tests for Arrow columnar copy in HostColumnarToG…
aec85ee 
…pu (NVIDIA#1665)

* Add Data source v2 test classes

Signed-off-by: Thomas Graves <tgraves@nvidia.com>

* update v2 source testing

* fix batch num rows and logging

* update the numberin batch

* Fix issue with reading booleans from ArrowColumnVectors and add more
tests

* move test file so pytest regex pick it up

* add comments

* fix line length

Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlowe jlowe jlowe approved these changes

Assignees

@tgravescs tgravescs

Labels
test Only impacts tests
Projects
None yet
Milestone
Feb 1 - Feb 12
Development

Successfully merging this pull request may close these issues.

[FEA] Automate integration tests for Arrow HostColumnToGpu functionality
2 participants
@tgravescs @jlowe