Cannot create Dask Client to a dask-scheduler with UCX protocol

[randerzander]

With latest nightlies:

(rapids) rgelhausen@rl-dgx-d19-u08-rapids-dgx102:~rapids-queries/q18$ conda list | grep ucx
ucx                       1.7.0+g9d06c3a       cuda10.1_0    rapidsai-nightly
ucx-proc                  1.0.0                       gpu    rapidsai-nightly
ucx-py                    0.13.0a200318+g9d06c3a         py37_76    rapidsai-nightly

I'm no longer able to create a Dask Client object to a scheduler running with UCX protocol:

# url - ucx://10.150.162.155
Traceback (most recent call last):
  File "/home/rgelhausen/tests/test_cluster.py", line 10, in <module>
    client = Client(url+':8786')
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py", line 721, in __init__
    self.start(timeout=timeout)
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py", line 894, in start
    sync(self.loop, self._start, **kwargs)
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/utils.py", line 348, in sync
    raise exc.with_traceback(tb)
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/utils.py", line 332, in f
    result[0] = yield future
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py", line 989, in _start
    await self._ensure_connected(timeout=timeout)
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/client.py", line 1073, in _ensure_connected
    assert msg[0]["op"] == "stream-start"
AssertionError

distributed.scheduler - INFO - Receive client connection: Client-fb038d70-6922-11ea-a737-d8c497cf5f77
distributed.scheduler - INFO - Close client connection: Client-fb038d70-6922-11ea-a737-d8c497cf5f77
distributed.core - ERROR - tuple indices must be integers or slices, not str
Traceback (most recent call last):
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/core.py", line 412, in handle_comm
    result = await result
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/scheduler.py", line 2502, in add_client
    versions,
  File "/home/rgelhausen/conda/envs/rapids/lib/python3.7/site-packages/distributed/versions.py", line 125, in error_message
    node_packages[node]["python"] = info["host"]["python"]
TypeError: tuple indices must be integers or slices, not str

The scheduler was started as:

DASK_UCX__RMM_POOL_SIZE=1GB DASK_UCX__ENABLE_INFINIBAND="False" DASK_UCX__ENABLE_NVLINK="True" dask-scheduler --protocol ucx

[quasiben]

I tried reproducing with a clean env today and was unable. I believe @randerzander is assessing his conda env

[mrocklin]

This corresponds to a change upstream. I suspect that not all of the components of the Dask cluster have been fully updated to master.

…

On Wed, Mar 18, 2020, 7:36 AM Benjamin Zaitlen ***@***.***> wrote: I tried reproducing with a clean env today and was unable. I believe @randerzander <https://github.com/randerzander> is assessing his conda env — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#459 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTD5GFEC6JKTPW7MNLDRIDL5FANCNFSM4LOSR6TQ> .

[randerzander]

Resolved after rebuilding environments. Thanks!

[mrocklin]

FYI @jrbourbeau

…

On Wed, Mar 18, 2020 at 10:14 AM Randy Gelhausen ***@***.***> wrote: Closed #459 <#459>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#459 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTFV4JAZRSXFZVVRND3RID6QXANCNFSM4LOSR6TQ> .

[jakirkham]

What change are you referring to Matt? 🙂

[jrbourbeau]

We added the Python version to the list of version checks in dask/distributed#3567. I suspect TypeError: tuple indices must be integers or slices, not str is coming from changing the output of get_system_info to be a dictionary instead of a tuple in that PR. Glad to hear rebuilding envs fixed things. Do let me know if this pops up again