[FEA] Improve PTDS performance

[rongou]

Is your feature request related to a problem? Please describe.
CUDA per-thread default stream (PTDS) is enabled for the plugin, but benchmark results haven't shown a big jump compared to the legacy default stream. Need to figure out why and try to improve the performance.

Describe the solution you'd like
At the moment there is no clear solution, but here are some ideas:

• Get rid of cudaDeviceSynchronize calls (reduce cudaDeviceSynchronize calls thrust#1255)
• Profiling using a smaller dataset on the desktop to better understand performance bottlenecks
• Benchmark with the shuffle manager and UCX enabled to ameliorate the effects of I/O
• Fix synchronous cudaMemcpy and cudaMemset calls ([REVIEW] call cudaMemcpyAsync/cudaMemsetAsync in JNI [skip ci] rapidsai/cudf#5913)
• Improve CUDA event handling in RMM's pool_memory_resource
• Reduce GPU memory fragmentation with multiple streams
• Better support spilling, where memory is freed from a different thread than the one originally allocated it
• Fix cudaErrorIllegalAddress errors (see [BUG]cudaErrorIllegalAddress an illegal memory access was encountered rapidsai/rmm#563)
• (Nice to have) Remove unnecessary cudaStreamSynchronize calls

Describe alternatives you've considered
It's possible some of the benchmark queries are too I/O bound that increasing GPU concurrency does not help with reducing the wall clock time.

Additional context
Original issue to enable PTDS: #15

[harrism]

but benchmark results haven't shown a big jump compared to the legacy default stream.

There's an important option you really should consider: a single stream might not be the bottleneck?

I think your first checkbox is crucial (simple standalone benchmark profiling).

[JustPlay]

@rongou

"but benchmark results haven't shown a big jump compared to the legacy default stream"

what benchmark? and how much?

thanks

[rongou]

We are using TPCx-BB. When I/O is very saturated, PTDS is only slightly faster, but probably not statistically significant. On a single GPU there are 10-20% improvements. Still investigating.

[rongou]

Tried with the shuffle manager enabled. Looks like there is more memory pressure/fragmentation, so had to increase the number of shuffle partitions/reduce gpu concurrency. Seems to perform better than without the shuffle manager, but it's not clear PTDS gives bigger gains.

[rongou]

This should be considered done.