[BUG] Importing dask_cuda results in import error with 'get_device_index_and_uuid`

[taureandyernv]

Describe the bug
When you import dask_cuda, it results in an ImportError saying that it cannot import name 'get_device_index_and_uuid' from 'distributed.diagnostics.nvml'

Steps/Code to reproduce bug
When you run this:

import dask_cuda

Output:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/tmp/ipykernel_186/2423259624.py in <module>
----> 1 import dask_cuda

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cuda/__init__.py in <module>
     10 
     11 from ._version import get_versions
---> 12 from .cuda_worker import CUDAWorker
     13 from .explicit_comms.dataframe.shuffle import get_rearrange_by_column_tasks_wrapper
     14 from .local_cuda_cluster import LocalCUDACluster

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cuda/cuda_worker.py in <module>
     19 
     20 from .device_host_file import DeviceHostFile
---> 21 from .initialize import initialize
     22 from .proxify_host_file import ProxifyHostFile
     23 from .utils import (

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cuda/initialize.py in <module>
      7 import dask
      8 import distributed.comm.ucx
----> 9 from distributed.diagnostics.nvml import get_device_index_and_uuid, has_cuda_context
     10 
     11 from .utils import get_ucx_config

ImportError: cannot import name 'get_device_index_and_uuid' from 'distributed.diagnostics.nvml' (/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/diagnostics/nvml.py)

Expected behavior
The import will work and dask_cuda will start

Environment overview (please complete the following information)
Affects all tested nightly docker containers (dev-nightly, nightly) in all OS (rocky, ubuntu, coentos7). core is not tested.

dask                      2022.9.3a221014 py_gbac0db81_25    dask/label/dev
dask-core                 2022.9.3a221014 py_gc4d35f551_25    dask/label/dev
dask-cuda                 22.12.00a221118 py38_gf11abe3_29    rapidsai-nightly
dask-cudf                 22.12.00a221118 cuda_11_py38_gcc4b4dd27c_281    rapidsai-nightly
distributed               2022.9.3a221014 py_gbac0db81_25    dask/label/dev

Additional Information
Found doing notebooks tests on the 10 minutes notebook

[taureandyernv]

Added distributed as per @pentschev's request

[wence-]

distributed 2022.9.3a221014 py_gbac0db81_25 dask/label/dev

This is old. Why is it being picked up? In fact, all of the dask "nightly" components are over a month behind.

[vyasr]

I am suspecting that #12231 is the culprit.

[vyasr]

Updating: I was wrong #12231 was an unrelated bug.

[shwina]

@taureandyernv - is this still an issue? I suspect not, as I'm able to import dask_cuda without problems. Please feel free to reopen if you are still seeing this behaviour.

[vyasr]

IIRC this issue was traced back to issues updating our containers that went way up the stack (container build failures stemming from a missing xgboost or so). Definitely out of scope for this repo.