[Build] error: more than one operator "+" matches these operands

[cyrusbehr]

Describe the issue

When trying to compile the library from source with support for CUDA 12.2, it fails with error:

error: more than one operator "+" matches these operands:
            function "operator+(const __half &, const __half &)" (declared at line 606 of /usr/local/cuda/targets/x86_64-linux/include/cuda_fp16.hpp)
            function "onnxruntime::cuda::operator+(const half &, const half &)" (declared at line 23)
            operand types are: const __half + const __half
    r.x = lh.x + rh.x;
               ^

Using latest commit on main branch: eeef15788804b0217d46ad6d1c2744585ad98d0c.

Cuda 12.2
nvcc version: Cuda compilation tools, release 12.2, V12.2.91
gcc (Ubuntu 11.4.0-2ubuntu1~20.04) 11.4.0

Urgency

Normal

Target platform

Ubuntu 20.04.5 LTS, 11th Gen Intel(R) Core(TM) i9-11900H @ 2.50GHz

Build script

export CFLAGS="-O3"
export CXXFLAGS="${CFLAGS}"

wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/tars/TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz
tar -zxvf TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0.tar.gz
export TENSORRT_HOME=$(readlink -f ./TensorRT-8.6.1.6)
export CUDA_HOME=/usr/local/cuda
export CUDNN_HOME=/usr/lib/x86_64-linux-gnu

git clone --recursive https://github.com/Microsoft/onnxruntime.git --depth 1
cd onnxruntime 

git checkout main

./build.sh --parallel 15 --use_cuda --cuda_home $CUDA_HOME --use_tensorrt --tensorrt_home $TENSORRT_HOME --allow_running_as_root --config RelWithDebInfo --build_shared_lib --compile_no_warning_as_error --skip_tests

Error / output

[ 51%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda_obj.dir/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/math/softmax_impl.cu.o
[ 51%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda_obj.dir/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/math/topk_impl_f16.cu.o
[ 51%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda_obj.dir/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/math/topk_impl_f32.cu.o
[ 51%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda_obj.dir/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/math/topk_impl_f64.cu.o
/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/cu_inc/common.cuh(80): error: more than one operator "+" matches these operands:
            function "operator+(const __half &, const __half &)" (declared at line 606 of /usr/local/cuda/targets/x86_64-linux/include/cuda_fp16.hpp)
            function "onnxruntime::cuda::operator+(const half &, const half &)" (declared at line 23)
            operand types are: const __half + const __half
    r.x = lh.x + rh.x;
               ^

/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/cu_inc/common.cuh(81): error: more than one operator "+" matches these operands:
            function "operator+(const __half &, const __half &)" (declared at line 606 of /usr/local/cuda/targets/x86_64-linux/include/cuda_fp16.hpp)
            function "onnxruntime::cuda::operator+(const half &, const half &)" (declared at line 23)
            operand types are: const __half + const __half
    r.y = lh.y + rh.y;
               ^

/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/cu_inc/common.cuh(87): error: more than one operator "-" matches these operands:
            function "operator-(const __half &, const __half &)" (declared at line 612 of /usr/local/cuda/targets/x86_64-linux/include/cuda_fp16.hpp)
            function "onnxruntime::cuda::operator-(const half &, const half &)" (declared at line 24)
            operand types are: const __half - const __half
    r.x = lh.x - rh.x;
               ^

/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/cu_inc/common.cuh(88): error: more than one operator "-" matches these operands:
            function "operator-(const __half &, const __half &)" (declared at line 612 of /usr/local/cuda/targets/x86_64-linux/include/cuda_fp16.hpp)
            function "onnxruntime::cuda::operator-(const half &, const half &)" (declared at line 24)
            operand types are: const __half - const __half
    r.y = lh.y - rh.y;
               ^

/home/cyrus/work/dartmouth-imaging/Eagle/build_scripts/scratch/onnxruntime/onnxruntime/core/providers/cuda/cu_inc/common.cuh(94): error: more than one operator "*" matches these operands:
            function "operator*(const __half &, const __half &)" (declared at line 618 of /usr/local/cuda/targets/x86_64-linux/include/cuda_fp16.hpp)
            function "onnxruntime::cuda::operator*(const half &, const half &)" (declared at line 25)

Visual Studio Version

No response

GCC / Compiler Version

No response

[sdpython]

It seems a duplicate of this issue #16713. It seems to work fine with 12.2.

[cyrusbehr]

I think you mean to say it works fine with CUDA 12.1.
But ok sounds good, will downgrade from CUDA 12.2 to CUDA 12.1 for now.

[cyrusbehr]

Can confirm that downgrading to CUDA 12.1 solved the issue.

[agirault]

For reference: fixed in cee050f (part #16831, merged as 7b9d1f1 and included since release v1.16.0)