add ctest memcheck using cuda-sanitizer (#9414)

addresses part of #904 and pre-requisite for rapidsai/ops#1462
- Adds `cuda-sanitizer` as memcheck tool in ctest.
- Adds environment variable `GTEST_CUDF_RMM_MODE` in gtests fixture

This PR enables the unit tests to run under `compute-sanitizer --tool memcheck` in gpuCI nightly or weekly builds.






The main issue pending in this PR is to find suitable way for specifying **"--rmm_mode=cuda"** only in memcheck runs.
`ctest` doesn't allow passing arguments to tests yet ([ctest feature request](https://gitlab.kitware.com/cmake/cmake/-/issues/20470)).

So, options are 
1. environment variable in cmake command (requires cmake rebuild_cache!)
    -    `export GTEST_CUDF_RMM_MODE=cuda; ninja rebuild_cache; ctest -T memcheck`
2. environment variable in ctest using [RunTests script](https://stackoverflow.com/a/54675911/1550940) (has limitations and fails sometimes)
3. ctest -S custom_script
4. add custom target in cmake instead of inbuilt ctest memcheck
5. Add environment variable in gtest for rmm mode. (Another easier option) *(This is implemented in this PR)*
    -    `export GTEST_CUDF_RMM_MODE=cuda; ctest -T memcheck`

Which method is most suitable? implemented [5]

Authors:
  - Karthikeyan (https://github.com/karthikeyann)

Approvers:
  - https://github.com/nvdbaranec
  - Robert Maynard (https://github.com/robertmaynard)
  - David Wendt (https://github.com/davidwendt)

URL: #9414

cpp/CMakeLists.txt

@@ -97,6 +97,11 @@ rapids_find_package(CUDAToolkit REQUIRED
     INSTALL_EXPORT_SET cudf-exports)
 include(cmake/Modules/ConfigureCUDA.cmake) # set other CUDA compilation flags
 
+# ctest cuda memcheck
+find_program(CUDA_SANITIZER compute-sanitizer)
+set(MEMORYCHECK_COMMAND ${CUDA_SANITIZER})
+set(MEMORYCHECK_TYPE CudaSanitizer)
+set(CUDA_SANITIZER_COMMAND_OPTIONS "--tool memcheck")
 
 ###################################################################################################
 # - dependencies ----------------------------------------------------------------------------------

cpp/include/cudf_test/base_fixture.hpp

@@ -266,16 +266,22 @@ inline std::shared_ptr<rmm::mr::device_memory_resource> create_memory_resource(
  *
  * Currently only supports 'rmm_mode' string parameter, which set the rmm
  * allocation mode. The default value of the parameter is 'pool'.
+ * Environment variable 'CUDF_TEST_RMM_MODE' can also be used to set the rmm
+ * allocation mode. If both are set, the value of 'rmm_mode' string parameter
+ * takes precedence.
  *
  * @return Parsing results in the form of unordered map
  */
 inline auto parse_cudf_test_opts(int argc, char** argv)
 {
   try {
     cxxopts::Options options(argv[0], " - cuDF tests command line options");
+    const char* env_rmm_mode = std::getenv("GTEST_CUDF_RMM_MODE");  // Overridden by CLI options
+    auto default_rmm_mode    = env_rmm_mode ? env_rmm_mode : "pool";
     options.allow_unrecognised_options().add_options()(
-      "rmm_mode", "RMM allocation mode", cxxopts::value<std::string>()->default_value("pool"));
-
+      "rmm_mode",
+      "RMM allocation mode",
+      cxxopts::value<std::string>()->default_value(default_rmm_mode));
     return options.parse(argc, argv);
   } catch (const cxxopts::OptionException& e) {
     CUDF_FAIL("Error parsing command line options");

cpp/tests/CMakeLists.txt

@@ -387,7 +387,7 @@ ConfigureTest(STRINGS_TEST
 
 ###################################################################################################
 # - structs test ----------------------------------------------------------------------------------
-ConfigureTest(STRUCTS_TEST 
+ConfigureTest(STRUCTS_TEST
     structs/structs_column_tests.cpp
     structs/utilities_tests.cpp
     )