Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]HostColumnarToGPU inefficient when only doing count() #1864

Closed
tgravescs opened this issue Mar 4, 2021 · 0 comments · Fixed by #1868
Closed

[BUG]HostColumnarToGPU inefficient when only doing count() #1864

tgravescs opened this issue Mar 4, 2021 · 0 comments · Fixed by #1868
Assignees
@tgravescs
Labels
bug Something isn't working performance A performance related task/issue
Milestone
Mar 1 - Mar 12

Comments

@tgravescs
Copy link
Collaborator

Describe the bug
If you are doing a count() operation, there are not any columns or data passed with the ColumnarBatch. The HostColumnarToGPU defaults to a row limit of 512 in this case, which is very inefficient. We should check for this case and make the limit much higher.

In one test scenario reading from a columnar source with I believe around 15 million rows, a simple count() took 2.1 minutes when it should have taken around 3 seconds.
@tgravescs tgravescs added bug Something isn't working performance A performance related task/issue labels Mar 4, 2021
@tgravescs tgravescs added this to the Mar 1 - Mar 12 milestone Mar 4, 2021
@tgravescs tgravescs self-assigned this Mar 4, 2021
@tgravescs tgravescs mentioned this issue Mar 4, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tgravescs tgravescs

Labels
bug Something isn't working performance A performance related task/issue
Projects
None yet
Milestone
Mar 1 - Mar 12
Development

Successfully merging a pull request may close this issue.

Increase row limit when doing count() for HostColumnarToGpu tgravescs/spark-rapids
1 participant
@tgravescs