Remaining Issues with HIP

[bmhan12]

Tasks from #778 that still remain:

• The shared TPLs are currently located at /usr/WS1/axom/libs/toss_4_x86_64_ib_cray_temp_dir/. The expected build_src.py default, /usr/WS1/axom/libs/toss_4_x86_64_ib_cray/, is a private user-only directory from someone that has left. @white238 is seeing if someone at LC can delete the directory.
• The installed examples (using-with-[cmake/blt/make]) fail and are disabled for hip. These examples are also failing for CUDA and were already disabled.
• There is a Fortran executable (quest_inout_interface_F_ex) and unit test (quest_inout_interface_mpi_F_test) that fails to compile with amdclang in a Debug build (File: src/axom/quest/examples/CMakeLists.txt). They are disabled under these conditions.
  • Suggestion by @kennyweiss convert condition to Fortran_COMPILER_FAMILY_IS_FOO type of call
• blt is not updated to develop with the hip changes. Still have issue with -lhip_runtime linkage error. This branch tracks the current PR and has latest blt with blt::hip.

EDIT:

• Fix spack configs to work with new uberenv/clingo concretizer

EDIT x2:

• mesh_tester executable does not run with hip policy (HIP policy for additional tests and examples #856 description for error)
  Done in PR Enable HIP policy for mesh_tester  #1151

EDIT x3:

• Investigate axom::Array usage with RAJA::ReduceSum (PR Comment link)

[rhornung67]

@bmhan12 you are using the latest BLT develop with amdclang++? what is being built when the -lhip_runtime error shows up?

[bmhan12]

@rhornung67 With the latest blt develop and hipcc, I've seen the error appear with any CMake target that depends on umpire or RAJA (e.g. "umpire_smoke_test").

[white238]

@rhornung67 With the latest blt develop and hipcc, I've seen it appear with any CMake target that depends on umpire or RAJA (e.g. "umpire_smoke_test").

This is more than likely because the Umpire release we are using is not exporting their hip_runtime target properly. This has been probably fixed in the new Umpire release. After both RAJA and Umpire have ironed out their BLT HIP support, I will help you with this.

We could also fake it. This makes a noop fake target in CMake:

add_library(hip_runtime INTERFACE)

Just stick it at the base of Axom somewhere with a giant TODO

[rhornung67]

Actually, I ran across that and the issue was some "very clever" stuff in camp. Here is the camp branch I am using to test the new BLT in RAJA: https://github.com/LLNL/camp/tree/raja-blt-update This isn't a complete fix and I am waiting on a response from Tom for something better. @bmhan12 you may want to try that camp branch and this RAJA branch that I have working now: https://github.com/LLNL/RAJA/tree/blt-update. Umpire develop has the new BLT stuff in it. Umpire develop and the camp and RAJA branches I linked here are all using the same commit of BLT develop.

[white238]

That makes more sense. Umpire is usually very good about exporting their targets properly.

[rhornung67]

camp's cmake stuff is too fancy(!) for me....

[kennyweiss]

Bump. Is this still an issue? It might be worth reinvestigating the open tasks on this issue