-
|
According to https://docs.nvidia.com/cuda/cuquantum/cutensornet/overview.html#slicing,
I'm running a QAOA circuit as described in https://arxiv.org/abs/2012.02430 (MaxCut problem for random graphs of degree 3). I have explicitly ensured that slicing is enabled, and have tweaked various slicing settings as shown in the following code snippet optimize_options = {
"samples": 5, # default is 0 (disabled)
"slicing": {
"disable_slicing": 0,
"memory_model": 1, # 0 is heuristic, 1 is cutensor (default)
"min_slices": 10, # default is 1
"slice_factor": 2, # default is 32
},
"cost_function": 0, # 0 for FLOPS (default), 1 for time
"reconfiguration": {
"num_iterations": 800, # default is 500; good values are within 500-1000
# Higher number means more time spent in reconfiguration, scales exponentially
"num_leaves": 10, # default is 8
},
}I split the contraction step into At 30 qubits, on a NVIDIA A100 40 GB, I constantly got this error, which means the contraction path is not efficient enough. But even so, shouldn't the slicing ensures that the contraction can run, albeit taking longer time (as a trade off for lower memory)? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 17 replies
-
|
Hello, Sorry for the delayed response. A couple of notes here:
# Approach I. Reuse Path
optimize_options = {"samples": 5, ...}
path, info = contract_path(expr, *operands, optimize=optimize)
output = contract(expr, *operands, optimize={'path': path})
# Approach 2. Direct
output = contract(expr, *operands, optimize=optimize)
import pickle
data = {'expr': expr,
'shapes': [o.shape for o in operands]}
with open('data.pickle', 'wb') as f:
pickle.dump(data, f) |
Beta Was this translation helpful? Give feedback.
-
|
I have published the code in https://github.com/rht/tensor_network_benchmark. It is runnable with This is how I specified the optimize options: https://github.com/rht/tensor_network_benchmark/blob/fba5ff33c55f36f8bef214bc1266510d9a3b7b0c/common_tn.py#L79-L88, and so, that the path finding is done twice should be ruled out. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing the script. Upon inspection of the logging, we noticed that the error is due to the number of elements of the largest intermediate tensor for that particular path exceeding int32_t limit. Such contraction path is currently not supported though we may be able to tune On a separate note, I noticed a few things in your sample:
|
Beta Was this translation helpful? Give feedback.
-
|
Upon further investigation, we realized that the number of elements exceeding int32_t was due to
Let us know if all the issues are resolved with the fixes. BTW, I think with the fix you won't need the custom |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! Both method fixed it, and I have tried several times without fixing the seed. This also means I have to redo all my existing |
Beta Was this translation helpful? Give feedback.
-
|
I read the The use case is that I want to to It seems safer for me to do this instead: def contract_path_return_network(...):
network = Network(...)
opt_info = network.contract_path(optimize=optimize)
return network, opt_infoLater on, I analyze |
Beta Was this translation helpful? Give feedback.
-
|
After the fix in #63 (reply in thread) and upgrading to 23.06, I occasionally encountered this error, with the same circuit I have been using: The error message is definitely different this time. But I wonder because this is due to me not gc-ing the |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the late reply. Here is the stdout and stderr output: https://gist.github.com/rht/e67ab461b24255653398676ccb2d5345. This is with According to https://docs.nvidia.com/cuda/cuquantum/latest/python/api/generated/cuquantum.NetworkOptions.html#cuquantum.NetworkOptions.memory_limit, 80% is the default? |
Beta Was this translation helpful? Give feedback.
-
|
The 80% default indeed refers to the total mem * 0.8. With BTW, your conversion from bytes to GB is not correct. The 16GB here equals 16*1024**3 bytes for our hardware. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, I have fixed the conversion factor.
I am able to consistently reproduce the OOM at this commit: rht/tensor_network_benchmark@c129894 (with MPS disabled, because MPS can't run on that QAOA ansatz). This is without deleting the computation output that still lives in the GPU, so it makes sense there is a leftover 4 GB from a previous computation (via
I tested. Deleting the |
Beta Was this translation helpful? Give feedback.
-
|
Just to confirm, you're seeing OOM when both It seems you're computing the expectation value as output and therefore the output itself should be tiny? My guess with Note that deleting output/intermediate or them going out of scope (both meaning reference count dropping to 0) will not trigger cupy to free that cache. The weird part is cupy should be able to reuse that 4GB cache if the reference count is really 0 with process synced (no explicit "free_all_blocks()" call needed). |
Beta Was this translation helpful? Give feedback.
-
Yes. I tested that I didn't run into OOM if I did |
Beta Was this translation helpful? Give feedback.
Upon further investigation, we realized that the number of elements exceeding int32_t was due to
slicinginformation not passed tooptimizeso thatcontractcall did not slice at all. Two simple fixes:optimize={"path": path}withoptimize={'path': path, 'slicing': info.slices}contract_pathandcontractcall, you can simply docontract(..., optimize=optimize_options). We internally pass both path and slicing.Let us know if all the issues are resolved with the fixes. BTW, I think with the fix you won't need the custom
optimize_options.