Microsoft.ML.OnnxRuntime.Gpu 1.18.0 not working with NVIDIA CUDA 11.6

[jacobilsoe]

Describe the issue

When using Microsoft.ML.OnnxRuntime.Gpu 1.18.0 with NVIDIA CUDA 11.6 on Windows I get an error. Microsoft.ML.OnnxRuntime.Gpu 1.17.3 and earlier versions work fine. According to https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html NVIDIA CUDA 11.6 should be compatible.
Error:

Microsoft.ML.OnnxRuntime.OnnxRuntimeException: [ErrorCode:RuntimeException] C:\a\_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1426 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "C:\onnx-issue\bin\Debug\net6.0\runtimes\win-x64\native\onnxruntime_providers_cuda.dll"

   at Microsoft.ML.OnnxRuntime.SessionOptions.AppendExecutionProvider_CUDA(Int32 deviceId)
   at Microsoft.ML.OnnxRuntime.SessionOptions.MakeSessionOptionWithCudaProvider(Int32 deviceId)

To reproduce

Run this line of C# code: using var gpuSessionOptions = SessionOptions.MakeSessionOptionWithCudaProvider();

Urgency

No response

Platform

Windows

OS Version

Windows 11 Pro - 23H2

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.18.0

ONNX Runtime API

C#

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.6

[yuslepukhin]

I think this is just a DLL loading issue which is related to the dependencies of onnxruntime_providers_cuda.dll.
Use dumpbin utility to find out the dependencies. Then use where command from the same environment to see if a particular dependency is found. Then add to the path or make sure it can be loaded otherwise.

See https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order

In general, the DLL above depends on CUDA libraries, and they need to be found either at the installation place (path) or in the process image folder.

[jacobilsoe]

Thanks. I did some more investigations and it seems that dependencies have changed between Microsoft.ML.OnnxRuntime.Gpu 1.17.3 and 1.18.0.

Dependencies of interest:

1.17.3:
cublasLt64_11.dll
cublas64_11.dll
cudnn64_8.dll
cufft64_10.dll
cudart64_110.dll

1.18.0:
cublasLt64_12.dll
cublas64_12.dll
cudnn64_8.dll
cufft64_11.dll
cudart64_12.dll

E.g. cublasLt64_12.dll is not supplied by CUDA 11.6. CUDA 11.6 supplies cublasLt64_11.dll.
So from what I can see Microsoft.ML.OnnxRuntime.Gpu 1.18.0 is no longer compatible with CUDA 11.6 on Windows.
So IMO either that breaking change should be documented or Microsoft.ML.OnnxRuntime.Gpu should revert to the previous dependencies.

[yufenglee]

1.18.0:
cublasLt64_12.dll
cublas64_12.dll
cudnn64_8.dll
cufft64_11.dll
cudart64_12.dll

ORT 1.18 GPU depends on CUDA 11.8. It should not be linked with cuda 12. @snnn , @pranavsharma.

[snnn]

I checked the onnxruntime_providers_cuda.dll downloaded from our 1.18.0 release page from onnxruntime-win-x64-gpu-1.18.0.zip. The DLL depends on the following CUDA DLLs

    cublas64_11.dll
    cublaslt64_11.dll
    cudart64_110.dll
    cudnn64_8.dll
    cufft64_10.dll

I don't see 12 there.

[snnn]

However, the package published to nuget.org is wrong. :-(

[1123612483]

Microsoft.ML.OnnxRuntime.Gpu 1.18.0 requires CUDA 11 and CUDA 12 to work properly.

[jacobilsoe]

So what is the conclusion here? 😃 That Microsoft.ML.OnnxRuntime.Gpu 1.18.0 requires CUDA 12 which means the NuGet package is correct but the zip file is wrong or is it the other way round?

[snnn]

I am sorry. The packages on nuget.org should use CUDA 11 instead. We uploaded CUDA 12 packages there by mistake. @jacobilsoe , which CUDA version do you need?

[jacobilsoe]

OK, thanks for clarifying. I need CUDA 11 compatibility. I will just wait for 1.18.1 to be released then. That one will target CUDA 11, right?

[snnn]

Yes

[jacobilsoe]

I can confirm that this is now fixed in 1.18.1