upgrade to nccl>=2.19 across RAPIDS

[jameslamb]

Description

wholegraph nightly conda-cpp-tests (amd64, rockylinux8, CUDA 11.4) have been failing with errors like this:

./WHOLEGRAPH_CSR_WEIGHTED_SAMPLE_WITHOUT_REPLACEMENT_TEST: symbol lookup error: /opt/conda/envs/test/bin/gtests/libwholegraph/../../../lib/libwholegraph.so: undefined symbol: ncclCommSplit
sh -c exec "$0" ./WHOLEMEMORY_HANDLE_TEST 
./WHOLEMEMORY_HANDLE_TEST: symbol lookup error: /opt/conda/envs/test/bin/gtests/libwholegraph/../../../lib/libwholegraph.so: undefined symbol: ncclCommSplit
sh -c exec "$0" ./GRAPH_APPEND_UNIQUE_TEST 

(build link)

ncclCommSplit was first introduced in nccl==2.18.1-1 (commit link) and is used unconditionally in wholegraph, but wholegraph (and other RAPIDS libraries) have a floor of nccl>=2.9.9.

In this particular job, nccl=2.10.3.1 is getting installed.

 + nccl                     2.10.3.1  hcad2f07_0                  rapidsai-nightly     125MB

We haven't noticed this in other jobs or other projects' CI because in all of those, nccl>=2.18.1 is getting installed.

In addition, pytorch 2.3 requires nccl>=2.19: #102 (comment)

Across RAPIDS, we should raise the floor on nccl to >=2.19.

Benefits of this work

• prevents a source of runtime errors when using RAPIDS ML libraries
• allows testing wholegraph in a CUDA 11.4 environment again

Acceptance Criteria

• all RAPIDS libraries with explicit nccl dependencies are pinned to nccl>=2.19

Approach

1. create a branch in https://github.com/rapidsai/shared-workflows adding the test job that's failing in wholegraph nightlies to the PR matrices
2. update wholegraph's pinnings, open a PR with CI pointed at that branch, and confirm that CI passes
3. point wholegraph CI back at branch-24.10 of shared-workflows, delete that shared-workflows branch
4. put up PRs with similar pins across all of the RAPIDS libraries with direct nccl dependencies
5. merge all those PRs (can be in any order)

Changes

Beta Give feedback

1. wholegraph (bump NCCL floor to 2.18.1.1, relax PyTorch pin wholegraph#218)
2. raft (bump NCCL floor to 2.18.1.1 raft#2443)
3. cugraph (bump NCCL floor to 2.18.1.1, include nccl.h where it's needed cugraph#4661)
4. cuvs (remove NCCL pins in build and test environments cuvs#341)
5. integration (bump NCCL floor to 2.18.1.1 integration#723)
6. cuopt (https://github.com/rapidsai/cuopt/pull/2017 ... not critical for RAPIDS 24.10)

[jakirkham]

In cuGraph, as part of the NumPy 2 upgrade ( rapidsai/cugraph#4615 (review) ), we noticed the following logic

# Starting from 2.2, PyTorch wheels depend on nvidia-nccl-cuxx>=2.19 wheel and
# dynamically link to NCCL. RAPIDS CUDA 11 CI images have an older NCCL version that
# might shadow the newer NCCL required by PyTorch during import (when importing
# `cupy` before `torch`).
if [[ "${NCCL_VERSION}" < "2.19" ]]; then
  PYTORCH_VER="2.1.0"
else
  PYTORCH_VER="2.3.0"
fi

Given this, am wondering if the minimum NCCL version should be 2.19?

cc @hcho3 (for awareness)

[jameslamb]

There are some competing requirements here:

• we want to pin pytorch>=2.3 to support numpy>=2
• pytorch>=2.3 requires NCCL >= 2.19
• raft's floor was just set to NCCL >=2.18.1.1, in bump NCCL floor to 2.18.1.1 raft#2443
• raft goes into code freeze for 24.10 in 4 hours
• raft is used widely, including in places that don't also use pytorch (like cuml)
• we generally want the floor for NCCL to be the same across RAPIDS libraries if possible

Given how close it is to 24.10 code freeze for raft, I don't think we should raise raft's floor further to 2.19 at this point. I think it's ok for cugraph to have a slightly higher floor than raft for the 24.10 release... in practice, both libraries have been building against much newer versions of NCCL for a while anyway. For example, look at the most recent successful nightly build of cugraph on 24.10... it used NCCL 2.22.3.1:

    nccl:                             2.22.3.1-hd961488_1                        conda-forge   

(build link)

Here's my proposal:

• 24.10:
  • raft stays pinned to nccl>=2.18.1.1 (because it's about to enter code freeze)
  • cugraph pins to nccl>=2.19 (to support pytorch>=2.3 and therefore numpy>2)
• 24.12
  • all RAPIDS libraries with NCCL pins to nccl>=2.19

@jakirkham @hcho3 what do you think?

[hcho3]

Sounds good to me

[jakirkham]

Think if we are planning to bump the minimum across the board, we should go ahead and get it done

Yes we are at code freeze and yes it is not great we are finding this change before then. However I think having the right intended behavior trumps that

Would add that a user not constraining NCCL likely gets a much newer version of NCCL anyways. This is also true of our own testing. So the lower bound is there to prevent unexpected solves and incompatible environments

One last note wholegraph also has a PyTorch dependency. So would need similar treatment to cuGraph

[jameslamb]

However I think having the right intended behavior trumps that

The issue you've found is at the intersection of PyTorch and NCCL. That's why I support bumping the floor up further to >=2.19 for the libraries that have PyTorch dependencies (and which, conveniently, are not in code freeze yet).

I thought that raft did not have a torch dependency... but was wrong about that 🙃

https://github.com/rapidsai/raft/blob/f37c41c54fc64a4e3689e5a61851ba3821800fee/python/pylibraft/pylibraft/common/outputs.py#L29-L36

So I guess we should do this there too.

[jameslamb]

@jakirkham in the interest of time, I started an internal thread about this.

[jakirkham]

Ok let's start with RAFT since that is going into code freeze

Then we can work on the rest

Edit: Didn't see your comment above when posting this. Will follow the internal thread

[jakirkham]

Thanks James! 🙏

Now that we have a PR for RAFT ( rapidsai/raft#2458 ), should we work on other cases (like Wholegraph)?

[jameslamb]

Yes, I'll do that right now.

[jameslamb]

Think all the updates we want now have associated PRs:

• wholegraph: bump NCCL floor to 2.19 wholegraph#223
• raft: bump NCCL floor to 2.19 raft#2458
• cugraph: should be done together with NumPy 2 updates (Remove NumPy <2 pin cugraph#4615 (comment), cc @hcho3 )
• cuvs: NCCL dropped as unneeded and later readded for a new feature needing NCCL
  • NCCL was removed in remove NCCL pins in build and test environments cuvs#341
  • NCCL was later readded for a new feature SNMG ANN cuvs#231 (comment)
• integration: bump NCCL floor to 2.19 integration#726
• cuopt: no action required
  • NCCL being removed in https://github.com/rapidsai/cuopt/pull/2017

[jakirkham]

Thanks James! 🙏

Also thank you for organizing that nice list 🙂

[jameslamb]

This is complete! Thanks very much for the help @jakirkham @hcho3 @alexbarghi-nv @cjnolet