-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add retry and SplitAndRetry support to AcceleratedColumnarToRowIterator #9088
Conversation
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
build |
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
build |
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuColumnarToRowExec.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuColumnarToRowExec.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
build |
numOutputRows += cb.numRows() | ||
if (cb.numRows() > 0) { | ||
numOutputRows += scb.numRows() | ||
if (scb.numRows() > 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry one more thing. If scb.numRows() <= 0
we probably should close it. Just to be on the safe side.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice catch. I will do it in a follow-up PR soon.
…#9102) This PR adds in retry support for more operations in GpuOutOfCoreSortIterator, including computing the split offset and bringing the data back to GPU to remove the projected columns. Besides, to keep being eager to close the input batches in the mergeSortAndClose function (introduced by #6931), instead of retrying the call to the whole mergeSortAndClose function, we retry the operations inside it, including bringing the data back to GPU, concatenating tables, sort the concatenated table and merging the input tables. It also covers a small followup change in GpuColumnToRowExec for PR #9088. --------- Signed-off-by: Firestarman <firestarmanllc@gmail.com>
fixes #8348
This PR adds retry and
SplitAndRetry
support toAcceleratedColumnarToRowIterator
.It will retry converting columns to rows by cudf when getting any oom exception.