Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix a minor potential issue when rebatching for GpuArrowEvalPythonExec #5995

Merged
merged 1 commit into from
Jul 14, 2022
Merged

Fix a minor potential issue when rebatching for GpuArrowEvalPythonExec #5995

merged 1 commit into from
Jul 14, 2022

Conversation

wbo4958
Copy link
Collaborator

@wbo4958 wbo4958 commented Jul 13, 2022

There is a chance that the row number of the combined batches is
just a multiple of batch size, which results in returning this huge
ColumnBatch directly.

-    if (rc % targetRoundoff == 0 || rc < targetRoundoff) {
+    if (rc <= targetRoundoff) {
       return combined
     }

@wbo4958 wbo4958 requested a review from firestarman July 13, 2022 03:44
@wbo4958
Fix a minor potential issue when rebatching for GpuArrowEvalPythonExec
f09249b 
There is a chance that the row number of the combined batches is
just a multiple of batch size, which results in returning this huge
ColumnBatch directly.

Signed-off-by: Bobby Wang <wbo4958@gmail.com>
@wbo4958
Copy link
Collaborator Author

wbo4958 commented Jul 14, 2022

build

gerashegalov
gerashegalov approved these changes Jul 14, 2022
View reviewed changes
Copy link
Collaborator

@gerashegalov gerashegalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pxLi
Copy link
Collaborator

pxLi commented Jul 14, 2022

build

1 similar comment
@wbo4958
Copy link
Collaborator Author

wbo4958 commented Jul 14, 2022

build

@pxLi pxLi added the bug Something isn't working label Jul 14, 2022
@wbo4958 wbo4958 merged commit 4a313e7 into NVIDIA:branch-22.08 Jul 14, 2022
@wbo4958 wbo4958 deleted the gpu-map-in-pandas branch July 14, 2022 06:29
wbo4958 added a commit to wbo4958/spark-rapids that referenced this pull request Nov 4, 2022
@wbo4958
Revert "Fix a minor potential issue when rebatching for GpuArrowEvalP…
72e27d7 
…ythonExec (NVIDIA#5995)"

This reverts commit 4a313e7.
@wbo4958 wbo4958 mentioned this pull request Nov 4, 2022
Merged
wbo4958 added a commit to wbo4958/spark-rapids that referenced this pull request Nov 4, 2022
@wbo4958
Revert "Fix a minor potential issue when rebatching for GpuArrowEvalP…
188e70c 
…ythonExec (NVIDIA#5995)"

This reverts commit 4a313e7.

Signed-off-by: Bobby Wang <wbo4958@gmail.com>
wbo4958 added a commit that referenced this pull request Nov 4, 2022
@wbo4958
Revert "Fix a minor potential issue when rebatching for GpuArrowEvalP…
02f87d2 
…ythonExec (#5995)" (#7006)

This reverts commit 4a313e7.

Signed-off-by: Bobby Wang <wbo4958@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlowe jlowe jlowe approved these changes

@gerashegalov gerashegalov gerashegalov approved these changes

@firestarman firestarman Awaiting requested review from firestarman

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wbo4958 @pxLi @jlowe @gerashegalov