[FEA] Audit the existing code to see what is and what is not covered by the retry framework

[revans2]

Is your feature request related to a problem? Please describe.
We have covered a lot of operators in the retry framework, but we have not covered everything. We also don't fully have support for split and retry everywhere. This is to go through all of the code for the various operators to find If the code allocated GPU memory

1. is that allocation covered by the retry framework
2. if so does the retry code support a split and retry
3. If there is no split and retry coverage would the algorithms support this (Some algorithms inherently cannot without a substantial change to how the algorithms work).

Then once we have all of this information we can file follow on issues to cover anything that was missed.

NOTE that all expressions should already be covered (at least everything except for Random with is the only non-deterministic expression that we support)

[revans2]

Looking at Spark 3.3.0 I see the following Exec that we need to look through and deal with retry for. I will go through this list update it with either follow on issues to add full retry support, or an indication that it is done.

• GpuTopN [FEA] Support Split and Retry for GpuTopN #8310 and [FEA] Add retry support to GpuSorter.fullySortBatch #8352
• GpuLocalLimitExec [FEA] Support Retry for GpuBaseLimitExec #8315
• GpuGlobalLimitExec [FEA] Support Retry for GpuBaseLimitExec #8315
• GpuRangeExec [FEA] SupportSplitAndRetry for GpuRangeExec #8314
• GpuFilterExec [FEA] Allow checkpoint and restart on non-deterministic expressions #7865
• GpuProjectExec [FEA] Allow checkpoint and restart on non-deterministic expressions #7865 and [FEA] Support split and retry for project #7866
• GpuCachedDoublePassWindowExec [FEA] Update GpuCachedDoublePassWindowIterator to use OOM retry framework #8217
• GpuColumnarToRowExec [FEA] Add retry and SplitAndRetry support to AcceleratedColumnarToRowIterator #8348
• GpuRowToColumnarExec [FEA] Add Retry support to GeneratedInternalRowToCudfRowIterator #8349 and [FEA] Add retry support to RowToColumnarIterator #8350
• GpuSortExec [FEA] Add retry support to GpuSorter.fullySortBatch #8352 and [FEA] Add retry support to GpuOutOfCoreSortIterator.mergeSortEnoughToOutput #8351 (The second one is more important because the other one is configured off, except as a part of GpuTopN)
• GpuHashAggregateExec So many of them for performance and better algorithms, but the ones we really care about is [FEA] Implement a heuristic to split a project's input based on output and add to hash aggregate #8382
• GpuExpandExec [FEA] Support spill+retry for GpuExpandExec #8461
• GpuShuffleCoalesceExec [FEA] Add retry to GpuShuffleCoalesceExec #8501
• GpuShuffledHashJoinExec [FEA] add retry to GpuHashJoin.filterNulls #8347
• GpuBroadcastHashJoinExec [FEA] add retry to GpuHashJoin.filterNulls #8347
• GpuRunningWindowExec [FEA] Update GpuRunningWindowExec for SplitAndRetryOOM #8343
• GpuBroadcastNestedLoopJoinExec [FEA] Add retry to creating the gather maps for CrossJoinIterator #8344 and [FEA] Add retry to ConditionalNestedLoopJoinIterator #8345
• GpuCartesianProductExec [FEA] Add retry to creating the gather maps for CrossJoinIterator #8344 and [FEA] Add retry to ConditionalNestedLoopJoinIterator #8345
• GpuProjectAstExec (This is really only used for testing, but...) [FEA] Support Split And Retry for GpuProjectAstExec #8311
• GpuSampleExec [FEA] Support SplitAndRetry for GpuSampleExec #8312
• GpuFastSampleExec [FEA] Support SplitAndRetry for GpuFastSampleExec #8313
• HostColumnarToGpu [FEA] Support Retry on HostColumnarToGpu #8326
• GpuHiveTableScanExec [FEA] Support retry for GpuHiveTableScanExec #8456
• GpuInMemoryTableScanExec [FEA] Add retry to ParquetCachedBatchSerializer #8459
• GpuGenerateExec [FEA] Finish retry and retryAndSplit support in GpuGenerateExec #8506
• GpuRapidsProcessDeltaMergeJoinExec [FEA] Add retry for GpuRapidsProcessDeltaMergeJoinExec #8505
• GpuDeltaInvariantCheckerExec [FEA] Add retry for GpuDeltaInvariantCheckerExec #8504
• GpuShuffleExchangeExec [FEA] Add retry to GpuShuffleExchangeExec #8502
• GpuDataWritingCommandExec
• GpuPythonMapInArrowExec
• GpuArrowEvalPythonExec
• GpuAggregateInPandasExec
• GpuFlatMapGroupsInPandasExec
• GpuFlatMapCoGroupsInPandasExec
• GpuMapInPandasExec
• GpuWindowInPandasExec
• GpuRapidsDeltaWriteExec This exec is a node we had to add in order to get access to the child plan when AQE is enabled. IT doesn't do anything interesting, so we shouldn't need a retry here.
• GpuFileSourceScanExec
• GpuBatchScanExec
• GpuExecutedCommandExec
• GpuCustomShuffleReaderExec
• GpuRunnableCommandExec
• GpuBroadcastToRowExec Just copies data to the CPU.
• GpuWindowExec Instead of updating GpuWidnowExec to support SplitAndRetry generally I think it would be better for us to support more window optimizations so we don't need the entire window in a single batch and add retry/split and retry to them.
• GpuCoalesceBatches DONE by Use withRetry in GpuCoalesceBatches #7852
• GpuSubqueryBroadcastExec CPU only.
• GpuBroadcastExchangeExec does not allocate GPU memory. Only copies data to the CPU. When the data goes back to the GPU it is in a Broadcast and the exec reading that needs to handle that case.
• GpuBringBackToHost does not allocate GPU memory. Only copies data to the CPU.
• GpuCoalesceExec Coalesce is just an RDD level operation. It does not process any data.
• GpuUnionExec Union is just a logical operation. It only updated some metrics and does not actually process any data.

[firestarman]

Hi, I saw most of the follow on issues don't have the roadmap or prioritiy specified.

I will work on these issues and it would be good to know their roadmap and prioritiy. Or just following the order of the list here is good enough ?

[revans2]

@firestarman I think the order here is fine. Be aware that the non-determinisitc expressions for Project and Filter might be a little more difficult and so it is okay if you want to try and skip it for now.