can't build using OpenMPI 1.5.5 and Boost 1.65

[mteimoori]

I used the following command to configure the CMake

cmake -G Ninja . -DCMAKE_PREFIX_PATH="/path/hip/build/lib/" -DHIPSYCL_PLATFORM=cuda -DHIPSYCL_GPU_ARCH=sm_35 -DCMAKE_INSTALL_PREFIX="/path/celerity-build" -DBOOST_ROOT=/path/boost-build -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_CXX_FLAGS="-std=c++14"  -DCMAKE_SYCL_FLAGS="--gcc-toolchain=/path/gcc/7.3.0"

After everything was successfully configured I tried to build it using:
ninja install
but I got the following error:

FAILED: CMakeFiles/celerity_runtime.dir/src/device_queue.cc.o
/path/hip/build/bin/syclcc-clang  -Iinclude -Ivendor -I/path/boost-build/include -I/path/openmpi/gcc/openmpi-1.5.5/@sys/include -Ispdlog-src/include --gcc-toolchain=/path/gcc/7.3.0 -fdiagnostics-color=always -O3 -DNDEBUG   -Wall -Wextra -Wno-unused-parameter -MD -MT CMakeFiles/celerity_runtime.dir/src/device_queue.cc.o -MF CMakeFiles/celerity_runtime.dir/src/device_queue.cc.o.d -o CMakeFiles/celerity_runtime.dir/src/device_queue.cc.o -c src/device_queue.cc
syclcc fatal error: Required command line argument --hipsycl-gpu-arch or environment variable HIPSYCL_GPU_ARCH not specified

I managed to fix this issue by setting env var:
export HIPSYCL_GPU_ARCH=sm_35
but now I have a different error after running ninja install:

FAILED: CMakeFiles/celerity_runtime.dir/src/config.cc.o
/path/hip/build/bin/syclcc-clang  -Iinclude -Ivendor -I/path/boost-build/include -I/afs/math.tu-berlin.de/software/openmpi/gcc/openmpi-1.5.5/@sys/include -Ispdlog-src/include --gcc-toolchain=/afs/math/software/gcc/7.3.0 -fdiagnostics-color=always -O3 -DNDEBUG   -Wall -Wextra -Wno-unused-parameter -MD -MT CMakeFiles/celerity_runtime.dir/src/config.cc.o -MF CMakeFiles/celerity_runtime.dir/src/config.cc.o.d -o CMakeFiles/celerity_runtime.dir/src/config.cc.o -c src/config.cc
In file included from src/config.cc:11:
In file included from include/workaround.h:3:
In file included from /net/path/hip/build/bin/../include/CL/sycl.hpp:35:
In file included from /net/path/hip/build/bin/../include/CL/sycl/backend/backend.hpp:51:
In file included from /net/path/hip/build/bin/../include/hipSYCL/hip/hip_runtime.h:58:
In file included from /net/path/hip/build/bin/../include/hipSYCL/hip/nvcc_detail/hip_runtime.h:28:
In file included from /net/path/hip/build/bin/../include/hipSYCL/hip/hip_runtime_api.h:325:
/net/path/hip/build/bin/../include/hipSYCL/hip/nvcc_detail/hip_runtime_api.h:886:9: warning: variable 'cdattr' is used uninitialized whenever switch default is taken [-Wsometimes-uninitialized]
        default:
        ^~~~~~~
/net/path/hip/build/bin/../include/hipSYCL/hip/nvcc_detail/hip_runtime_api.h:891:41: note: uninitialized use occurs here
    cerror = cudaDeviceGetAttribute(pi, cdattr, device);
                                        ^~~~~~
/net/path/hip/build/bin/../include/hipSYCL/hip/nvcc_detail/hip_runtime_api.h:786:5: note: variable 'cdattr' is declared here
    enum cudaDeviceAttr cdattr;
    ^
src/config.cc:102:41: error: use of undeclared identifier 'OMPI_COMM_TYPE_HOST'
                                MPI_Comm_split_type(MPI_COMM_WORLD, SPLIT_TYPE, 0, MPI_INFO_NULL, &node_comm);
                                                                    ^
src/config.cc:93:20: note: expanded from macro 'SPLIT_TYPE'
#define SPLIT_TYPE OMPI_COMM_TYPE_HOST
                   ^
src/config.cc:137:5: error: function-like macro 'BOOST_PP_OVERLOAD' is not defined
#if WORKAROUND(HIPSYCL, 0)
    ^
include/workaround.h:38:58: note: expanded from macro 'WORKAROUND'
#define WORKAROUND(impl, ...) (WORKAROUND_##impl == 1 && _WA_CHECK_VERSION(__VA_ARGS__))
                                                         ^
include/workaround.h:33:32: note: expanded from macro '_WA_CHECK_VERSION'
#define _WA_CHECK_VERSION(...) BOOST_PP_OVERLOAD(_WA_CHECK_VERSION_, __VA_ARGS__)(__VA_ARGS__)
                               ^
1 warning and 2 errors generated when compiling for sm_35.

has anyone run into the same problem?

[psalz]

Hi, thanks for opening this issue! As I see it there's a couple of things here:

1. The CMake configuration problem might be due to some caching issues with the HIPSYCL_GPU_ARCH variable. You shouldn't have to set the corresponding environment variable, but if you didn't completely clean your build directory in-between CMake calls perhaps a stale (empty) version of the variable was being cached - I will look into that. By the way, unless you have other non-SYCL/Celerity targets in your CMake project, setting the C/CXX compilers to GCC won't have any effect, as using hipSYCL with the CUDA platform will always compile using Clang internally!

2. The error: use of undeclared identifier 'OMPI_COMM_TYPE_HOST' is likely due to the very old version of OpenMPI you're using (1.5.5), which was released in 2012. I've never tested Celerity with such an old version, and I can't say for sure that it will work even if we were to fix that missing macro (which is not strictly required). I suggest you try updating to a newer version of OpenMPI, preferably 4.x, as that's what we're testing Celerity with. I will however look into raising the minimum required version in our CMake config, so that this problem can be caught before building next time!

3. The last issue took me by surprise, since that is the kind of error I've been seeing with Boost 1.69+ (as mentioned on the website), and for which didn't have time to investigate the cause yet. So I just went back and tried compiling Celerity with various Boost versions; it looks like this issue also occurs with 1.65 and 1.66 - but not with 1.67 and 1.68. So as a quick fix I currently can only suggest to try 1.67 or 1.68, but I will take this opportunity to also try and figure out the actual issue (and hopefully push a fix soon).

[psalz]

The BOOST_PP_OVERLOAD should now be fixed (78142e4). To be clear, there is still some unrelated problems when trying to compile with Boost 1.69+, however boost 1.65-1.68 should all be fine now!

[mteimoori]

Hi, Thank you for the fix, it's working now with Boost 1.65, OpenMPI 4.0.1 and gcc 7.3.0 and I was able to build it. Although I tried it with a clean build directory again, I still had to set HIPSYCL_GPU_ARCH manually. Thanks a lot!

[psalz]

Okay, great! I've changed the title to reflect what was ultimately the problem (OpenMPI) in case somebody runs into this in the future. Also I'll look into that hipSYCL config variable!