[BUG] pinned_host_vector can cause abrupt program termination

[jlowe]

Describe the bug
pinned_host_vector can throw from within it's destructor, causing the application process to terminate abruptly. For example, this is causing abnormal termination of Spark executors when a GPU illegal access occurs during a Parquet read. Since the executor process is abruptly terminated via low-level abort(), there's no chance to convey a useful message from the driver to the executor otherwise try to handle at the application level the cudf::fatal_cuda_error exception that is being thrown.

Steps/Code to reproduce bug
Compile and run the following program:

#include <stdexcept>
#include <cudf/detail/utilities/pinned_host_vector.hpp>

__global__ void crash(int *x) {
  int *y = NULL;
  *x = y[12345678];
}

int main() {
  int *x;
  try {
    auto p = cudf::detail::pinned_host_vector<uint8_t>(4096);
    cudaMalloc(&x, sizeof(*x));
    crash<<<1,1>>>(x);
    cudaDeviceSynchronize();
  } catch (std::exception const& e) {
  }
  return 0;
}

which will produce the following output:

terminate called after throwing an instance of 'cudf::fatal_cuda_error'
  what():  Fatal CUDA error encountered at: /home/jlowe/src/spark-rapids-jni/target/libcudf-install/include/cudf/detail/utilities/pinned_host_vector.hpp:172: 700 cudaErrorIllegalAddress an illegal memory access was encountered
Aborted (core dumped)

Expected behavior
Exceptions should not be thrown from destructors and cause an application to be terminated abruptly with no chance for the application to shutdown gracefully.

[GregoryKimball]

I believe this example hits the pinned_allocator deallocate function here.

We could adjust deallocate to not throw, catch the exception in the destructor (which is currently empty), or perhaps choose another option that we haven't yet considered.

[jlowe]

We could adjust deallocate to not throw

This is what RMM's memory allocators do. Allocate functions always check, but deallocate is a debug assert that does nothing in a release build. See https://github.com/rapidsai/rmm/blob/branch-23.10/include/rmm/mr/device/cuda_memory_resource.hpp#L69-L88 as an example.