Is it possible to reduce path finding time?

[NathanCQC]

Hello,

I am contracting a circut with 25,000 tensors.

path, info = network.contract_path(optimize={'samples': n_samples, 'slicing': {'min_slices': max(max_n_slices, size)}})
In this regime, the above function takes about 8000 seconds and the contraction is 1.57s.

Is it possible to make the path finding less expensive?

Changing n samples from 1-10 has no effect on this time. Would you be able to provide some insight there?

Thanks alot.

[haidarazzam]

Thank you for reaching out to the cuQuantum team, we are delighted to help resolve your issue.
The pathfinding time is affected by the size of the network, the slicing, the number of samples and the number of threads. It is on average depending on the number of samples, for example using 1 thread 10 samples needs about 10 times the time of 1 sample, but 10 samples may requires almost same time as 1 sample if running on 10 or more threads.
What is the value of your min_slices? (Since the contraction time is too small, I would guess that no slicing is needed, usually we slice to make the contraction fit into the GPU but if the contraction time is small, it indirectly means that it is small contraction and fit into the GPU memory thus no need for slicing.
cuQuantum 23.06 (the version that is going to be released soon, has an automated option to reduce the pathfinder time based on the contraction time but at least there must be 1 sample (1 path) that need to be generated.
Options to reduce the time include:

1. disable slicing
2. reduce reconfIters (default is 500-1000 and can be reduced down to 200 100 to 0) 'CUTENSORNET_CONTRACTION_OPTIMIZER_CONFIG_RECONFIG_NUM_ITERATIONS'
3. reduce reconfLeafs to 6 'CUTENSORNET_CONTRACTION_OPTIMIZER_CONFIG_RECONFIG_NUM_LEAVES'

Can you share your network?

[NathanCQC]

Yes I have done both and the performance is the same. My colleague @yapolyak has also done this.

It is most likely something wrong with the slurm input.

[NathanCQC]

These are the packages in side the container when I start a bash script inside it

Package Version

---

attrs 23.1.0
boltons 23.0.0
brotlipy 0.7.0
build 0.10.0
CacheControl 0.12.14
certifi 2023.5.7
cffi 1.15.1
charset-normalizer 2.0.4
cleo 2.0.1
conda 23.5.0
conda-content-trust 0.1.3
conda-package-handling 2.0.2
conda_package_streaming 0.7.0
crashtest 0.4.1
cryptography 39.0.1
cupy-cuda11x 12.1.0
cuquantum-python-cu11 23.3.0
custatevec-cu11 1.3.0
cutensor-cu11 1.7.0
cutensornet-cu11 2.1.0
Cython 0.29.33
distlib 0.3.6
dulwich 0.21.5
fastrlock 0.8.1
filelock 3.12.2
graphviz 0.20.1
html5lib 1.1
idna 3.4
importlib-metadata 6.6.0
installer 0.7.0
jaraco.classes 3.2.3
jeepney 0.8.0
Jinja2 3.1.2
jsonpatch 1.32
jsonpointer 2.1
jsonschema 4.17.3
keyring 23.13.1
lark-parser 0.12.0
lockfile 0.12.2
MarkupSafe 2.1.3
more-itertools 9.1.0
mpi4py 3.1.4
mpmath 1.3.0
msgpack 1.0.5
networkx 3.1
numpy 1.24.3
packaging 23.0
pandas 2.0.2
pexpect 4.8.0
pip 23.0.1
pkginfo 1.9.6
platformdirs 3.5.3
pluggy 1.0.0
poetry 1.5.1
poetry-core 1.6.1
poetry-plugin-export 1.4.0
ptyprocess 0.7.0
pycosat 0.6.4
pycparser 2.21
pyOpenSSL 23.0.0
pyproject_hooks 1.0.0
pyrsistent 0.19.3
PySocks 1.7.1
python-dateutil 2.8.2
pytket 1.16.0
pytz 2023.3
qwasm 1.0.1
rapidfuzz 2.15.1
requests 2.28.1
requests-toolbelt 1.0.0
ruamel.yaml 0.17.21
ruamel.yaml.clib 0.2.6
scipy 1.10.1
SecretStorage 3.3.3
setuptools 65.6.3
shellingham 1.5.0.post1
six 1.16.0
sympy 1.12
tomli 2.0.1
tomlkit 0.11.8
toolz 0.12.0
tqdm 4.65.0
trove-classifiers 2023.5.24
types-pkg-resources 0.1.3
typing_extensions 4.6.3
tzdata 2023.3
urllib3 1.26.15
virtualenv 20.23.0
webencodings 0.5.1
wheel 0.38.4
zipp 3.15.0
zstandard 0.19.0

[leofang]

Thanks for confirming, Nathan. I forgot that I designed python -m pip install . to pull our wheels from PyPI. So all is good regarding software env.

To your other reply earlier:

It is running on a single CPU thread I think.

Did you use htop or other tools on the compute node to confirm this?

[NathanCQC]

I am looking into this now.

I am also running from inside this container https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuquantum-appliance

But I think the problem is still the same.

[NathanCQC]

top is never over 100% (Much lower infact most of the time)

[yapolyak]

Can I jump in @yangcal and ask how do you control number of CPU threads you are running contract_path with? Are those logical threads of a single core or multiple cores? I presume hyperthreading might be off by default on Perlmutter. (I have Not run the example yet myself, let me try (now or tomorrow morning UK time)).

[yapolyak]

That's an interesting point @leofang - I will try this tomorrow! I am trying to use more than a single core, but according to top, cuQuantum is only using one... Perhaps I am doing something wrong with Slurm, although I don't think so...

[yapolyak]

Just to make sure - one should not use environment variables such as $OMP_NUM_THREADS?

[yapolyak]

Right I am telling Slurm to use 64 cores/threads (-c 64), explicitly telling cuQuantum to use 64 threads:

path, info = contract_path(*cuquantum_interleaved, optimize=OptimizerOptions(threads=64))

But it is still using only 100% CPU... I guess we need to ask Perlmutter people why that could be...

[yapolyak]

Just in case, I've set $OMP_NUM_THREADS to 64, still nothing. Tomorrow I will try a non-MPI installation. Many thanks and have a nice day!

[leofang]

As Yang alluded to in his other post, OMP_NUM_THREADS has no effect as we don't use OpenMP as the threading layer in cuTensorNet.

[yangcal]

Hello @NathanCQC @yapolyak ,

We've pinned down a couple of factors and hopefully that can resolve all the questions above:

1. Regarding the large performance difference in path-finding, turns out the cuquantum-python that we're testing with is our soon to-be-released 23.06 version (apologies here). We've added a few optimizations in our path-finder and that should be where the big difference is using my snippet. I tried the snippet again with 2303 and it was also stuck in the path-finding stage for at least tens of minutes.
2. For large tensor network like you have with n=7, you're encouraged to turn on hyper sampling via OptimizerOptions.samples(default is off). You can also adjust the number of threads via OptimizerOptions.threads or use the default half of cores. There is no need to specify OMP_NUM_THREADS for threading. Note that, threading kicks in after the first sample is finished. For your large network, the first sample itself is taking long time to complete and that's why you haven't observed the increased CPU utility. In 2306, there is also a new feature that further improves our path-finder when hyper sampling is on so stay tuned.

Hope that resolves all the questions above. We apologize for some negligence on our side and hope to exchange more feedbacks once our 2306 release is out.

[leofang]

btw 23.06 has been fully posted. Due to the complexity of our offering and release process, it may not be always possible to get all things done at once, and you did observe this (pip wheels were out earlier than other pieces). @NathanCQC @yapolyak Feel free to give it a shot and let us know how it goes! 🙂

[NathanCQC]

Wow, you guys turned this around very quickly. Thanks a lot. I will let you know and run the benchmarking script today.

I will also try the auto parallelism.

[yapolyak]

Great news! We will give it a go! Thank you very much for your work and support!

[NathanCQC]

I see x20 speed up. For the 7 site hubbard model.

[yangcal]

The x20 speedup was compared with the 1743s that you previously observed on a compute node? That amounts to ~80 seconds and is about on par with our internal benchmark.