Installation of cuQuantum on 4 A40 Datacenter GPUs for running qiskit circuits

[intelligi123]

Hi,
I have four A40 GPUs available which I need to make a cluster from and install cuQuantum in order to simulate some qiskit circuits to achieve parallel computing. I am getting information in bits and pieces but not sure how to start the task.

Can someone guide me here how to first make a cluster of GPUs ready for cuQuantum installation on high level please?

[mtjrider]

Hi, I have four A40 GPUs available which I need to make a cluster from and install cuQuantum …

What is the physical topology of the node? Are the A40s all on the same node? It would be helpful if you ran nvidia-smi topo -m and posted the output here.

Regarding the second question of installing cuQuantum, I assume you mean the cuQuantum Appliance?

[intelligi123]

VMs have been created using VSphere with allocation of one GPU to each. Being on one server, communication bandwidth wont be any issue as far as I think but you can suggest better;
My goal is to distribute a qiskit code incrementally (on all four VMs by adding one at a time) and notice performance up gradation.

If Its possible kindly share any link of step-by-step process of installing cuquantum appliance on multiple machines and running a sample code. It would be very helpful.

[mtjrider]

Hi @intelligi123 this is a great question. I know of no previous work/test with cuQuantum + vSphere.

Roughly speaking, this is the procedure I would follow:

1. Make sure that compiled CUDA examples run without issue and report sensible bandwidth measures between GPU and host.
2. Configure, install, and test MPI in your virtual machines. Confirmation might look like running a simple command: mpirun -n 2 bash -c 'echo hello from ${OMPI_COMM_WORLD_RANK}' (assuming open-mpi).
3. Using NCCL, you can evaluate the available bandwidth between GPUs.

These items will ensure that your computational environment is well defined before you approach using cuQuantum. Unfortunately, I have limited experience with vSphere and I won't be much help if you run into problems.

[intelligi123]

Thanks for the answer and please correct me if I am wrong, Isn't OPENMPI is builtin in cuQuantum Appliance? As on its homepage its given that for MPI applications, openMPI is already installed.

As far as I think, I need to install container (cuquantum appliance) on all four VMs and using MPI machinefile, distribute the code on vms

[mtjrider]

Hi @intelligi123

Yes, mpirun is available in the container.
One option is to pass VM host networking through to the container, and call mpirun from within the container.

To do this, you will need to configure each container's SSH so that they are all mutually reachable defining aliases and hostfiles.

For example:

# Assuming each VM has one GPU, create a detached container named after the VM's hostname
docker run -it --detach --name client_${client_hostname}_0 --gpus "'device=0'" \
    --network=host \
    --uts=host \
    --ipc=host \
    nvcr.io/nvidia/cuquantum-appliance:23.06

# Next, create an SSH config with aliases.
# $ cat ./config
# Host client_${client_hostname}_0
#     Hostname ${client_hostname}
#     Port ${client_hostname_port}
#     IdentityFile ~/path/to/id-file-in-container
#     StrictHostKeyChecking no
#     UserKnownHostsFile=/dev/null
# ...
# $ cat ./hostfile
# client_${client_hostname}_0
# ...
mkdir -p ./.sshc && ssh-keygen -t KEY_TYPE -C '' -N '' -f ./.sshc/id_KEY_TYPE && cat ./config > ./.sshc/config && cat ./hostfile > ./.sshc/hostfile  # note my use of `.sshc`
docker cp ./sshc client_${client_hostname}_0:/container-user/home/
...

From here, you need to ensure that the SSH service is actually running. This may require you reconfigure the daemon/service. These commands may be useful:

docker exec -i client_${client_hostname}_0 bash -c 'rm -rf CONTAINER_USER_HOME_DIR/.ssh'  # remove preexisting ssh directory
docker exec -i client_${client_hostname}_0 bash -c 'mv CONTAINER_USER_HOME_DIR/.sshc CONTAINER_USER_HOME_DIR/.ssh'
docker exec -i client_${client_hostname}_0 bash -c 'chown -R {user} CONTAINER_USER_HOME_DIR/.ssh'  # fix file permissions from copy
docker exec -i client_${client_hostname}_0 bash -c 'chmod 700 CONTAINER_USER_HOME_DIR/.ssh'  # fix file permissions from copy
docker exec -i client_${client_hostname}_0 bash -c 'chmod -R 600 CONTAINER_USER_HOME_DIR/.ssh'  # fix file permissions from copy
docker exec -i client_${client_hostname}_0 bash -c \"sed -i '/Port /c\Port HOSTNAME_PORT_GOES_HERE\' /etc/ssh/sshd_config\"  # change SSH port to match SSH config alias
docker exec -i client_${client_hostname}_0 bash -c \"sed -i '/#PermitRootLogin/c\PermitRootLogin yes\' /etc/ssh/sshd_config\"  # enable root ssh login
docker exec -i client_${client_hostname}_0 service ssh restart  # restart the service

Once this is done on each running docker container, you should be able to launch an MPI job from within the container by passing the command the appropriate hostfile (under CONTAINER_USER_HOME_DIR/.ssh/hostfile).

The aforementioned gets more complicated if the networking infrastructure connecting nodes is more complex (e.g.) requiring novel drivers, etc.. The method above has the benefit of providing finely tuned control over networking interfaces. If you don't need this, you can always look into something like mpi-operator.

Let me know if you try any of the above and run into issues. I'll add that there are many ways to accomplish the same thing.

EDIT: Removed my unfortunate use of ${hostname} as a variable when there is a command of the same name. I replaced hostname with client_hostname.

[mtjrider]

@intelligi123
Are the questions in this discussion addressed?
Do you have any other questions for us?

[mtjrider]

@intelligi123
Unless I hear from you, I will mark a representative response as the answer and lock this discussion. Of course, you are free to open another discussion or issue if you have other questions which need to be addressed.