Running the GHZ cusvaer example on multiple hosts, each has >= 1 GPU (Appliance 24.08)

[rht]

This is a continuation of #115, which worked on 23.10, but now I am testing on 24.08.

I got an output in the end, but with errors:

--------------------------------------------------------------------------
WARNING: Open MPI tried to bind a process but failed.  This is a
warning only; your job will continue, though performance may
be degraded.

  Local host:        d74d399cf4be
  Application name:  /opt/conda/envs/cuquantum-24.08/bin/python
  Error message:     failed to bind memory
  Location:          rtc_hwloc.c:447

--------------------------------------------------------------------------
[1727259491.503008] [d74d399cf4be:399  :0]     ucp_context.c:2119 UCX  INFO  Version 1.15.0 (loaded from /usr/local/ucx/lib/l
ibucp.so.0)
[1727259491.504759] [d74d399cf4be:400  :0]     ucp_context.c:2119 UCX  INFO  Version 1.15.0 (loaded from /usr/local/ucx/lib/l
ibucp.so.0)
[1727259492.931015] [d74d399cf4be:400  :0]       rdmacm_cm.c:949  UCX  DIAG  rdma_create_event_channel failed: No such device
[1727259492.931046] [d74d399cf4be:400  :0]      ucp_worker.c:1552 UCX  DIAG  failed to open CM on component rdmacm with statu
s Input/output error
[1727259492.931061] [d74d399cf4be:399  :0]       rdmacm_cm.c:949  UCX  DIAG  rdma_create_event_channel failed: No such device
[1727259492.931087] [d74d399cf4be:399  :0]      ucp_worker.c:1552 UCX  DIAG  failed to open CM on component rdmacm with statu
s Input/output error
[1727259492.931350] [d74d399cf4be:400  :0]          parser.c:2045 UCX  INFO  UCX_* env variables: UCX_TLS=all UCX_NET_DEVICES
=eth0 UCX_LOG_LEVEL=info
[1727259492.931371] [d74d399cf4be:399  :0]          parser.c:2045 UCX  INFO  UCX_* env variables: UCX_TLS=all UCX_NET_DEVICES
=eth0 UCX_LOG_LEVEL=info
[1727259492.932502] [d74d399cf4be:400  :0]     ucp_context.c:2119 UCX  INFO  Version 1.15.0 (loaded from /usr/local/ucx/lib/l
ibucp.so.0)
[1727259492.932502] [d74d399cf4be:399  :0]     ucp_context.c:2119 UCX  INFO  Version 1.15.0 (loaded from /usr/local/ucx/lib/l
ibucp.so.0)
[1727259492.933350] [d74d399cf4be:399  :0]      ucp_worker.c:1855 UCX  INFO    0x56253ad1ce10 self cfg#1 tag(self/memory cuda
_copy/cuda)
[1727259492.933362] [d74d399cf4be:400  :0]      ucp_worker.c:1855 UCX  INFO    0x5643ca7bcba0 self cfg#1 tag(self/memory cuda
_copy/cuda)
[1727259492.933444] [d74d399cf4be:399  :0]      ucp_worker.c:1855 UCX  INFO    0x56253ad1ce10 intra-node cfg#2 tag(sysv/memor
y tcp/eth0 cuda_ipc/cuda)
[1727259492.933451] [d74d399cf4be:400  :0]      ucp_worker.c:1855 UCX  INFO    0x5643ca7bcba0 intra-node cfg#2 tag(sysv/memor
y tcp/eth0 cuda_ipc/cuda)
[LOG_CAT_ML] You must specify a valid HCA device by setting:
-x HCOLL_MAIN_IB=<dev_name:port> or -x UCX_NET_DEVICES=<dev_name:port>.
If no device was specified for HCOLL (or the calling library), automatic device detection will be run.
In case of unfounded HCA device please contact your system administrator.
[LOG_CAT_ML] You must specify a valid HCA device by setting:
-x HCOLL_MAIN_IB=<dev_name:port> or -x UCX_NET_DEVICES=<dev_name:port>.
If no device was specified for HCOLL (or the calling library), automatic device detection will be run.
In case of unfounded HCA device please contact your system administrator.
[d74d399cf4be:00400] Error: coll_hcoll_module.c:310 - mca_coll_hcoll_comm_query() Hcol library init failed
[d74d399cf4be:00399] Error: coll_hcoll_module.c:310 - mca_coll_hcoll_comm_query() Hcol library init failed
mpi4py: rank: 0, size: 2
mpi4py: rank: 1, size: 2
Result: rank: 0, size: 2
{'11111111111111111111111111111': 500, '00000000000000000000000000000': 524}
Result: rank: 1, size: 2
[d74d399cf4be:00395] 1 more process has sent help message help-orte-odls-default.txt / memory not bound
[d74d399cf4be:00395] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

I modified the answer #115 (reply in thread) by replacing ens3 with eth0, because mpirun wants it to be either eth0 or lo. Otherwise, there would be an extra error message

[1727260043.662057] [d74d399cf4be:506  :0]     ucp_context.c:1177 UCX  WARN  network device 'ens3' is not available, please u
se one or more of: 'eth0'(tcp), 'lo'(tcp)
[1727260043.662060] [d74d399cf4be:505  :0]     ucp_context.c:1177 UCX  WARN  network device 'ens3' is not available, please u
se one or more of: 'eth0'(tcp), 'lo'(tcp)

nvidia-smi output after the script has completed.

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla V100-PCIE-16GB           Off | 00000001:00:00.0 Off |                  Off |
| N/A   31C    P0              36W / 250W |      0MiB / 16384MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

[ymagchi]

Hi @rht,

Thank you for the inquiry.
For 24.08 container images, HCOLL is enabled by default for optimal collective operations on systems using Mellanox InfiniBand adapters.
It can be disabled by adding an option --mca coll_hcoll_enable 0.

For more details, please refer to the "Software in the container" section of the NGC overview page: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuquantum-appliance.

[rht]

I see, thank you!