Actuator source term for wing simulation is only partially applied depending on number of MPI ranks

[ET14]

Bug description

The simulation setup is an "infinite" wing where the streamwise direction is x, the normal direction is z and the spanwise direction is y.
To model the "infinite wing" periodic BCs are applied in y and the wing span equals the domain width in y (Lymin=-8 < y < Lymax=+8). This setup should not exhibit any spanwise gradients (in terms of actuator source term and velocities). However, depending on the number of chosen MPI ranks this is not the case and the actuator source term is only applied from -8 < y < 0, but is missing between 0 < y < 8.

Steps to reproduce

1. This is the case configuration : epsilon4.0.zip
2. Run the case for a few seconds (e.g. T=16) with either 15 or 31 cores.
3. Both cases should not crash and run silently
4. Examine actuator_src_x and z term of the last plt file in a yz-plane at x=0 to observe the reported issue for 31 MPI ranks, but a correct distribution for 15 MPI ranks

Details:

1. Compiler used
   • GCC
   • LLVM
   • oneapi (Intel)
   • nvcc (NVIDIA)
   • rocm (AMD)
   • with MPI
   • other:
2. Operating system
   • Linux
   • OSX
   • Windows
   • other (do tell ;)):
3. Hardware:
   • CPU
   • GPU
4. Machine details ():
   please see attached slurm submission script
   @tonyinme replicated this issue on one of NREL's HPC systems, so the specific machine details might be secondary.

Expected behavior

The actuator source term should be applied from y=-8 to y=+8 and the y-velocity should be of order 10^(-14).
Note: This is not the most common AMR use case. In order to essentially obtain a 2D wing simulation which uses grid refinement the number of grid points in y is Ny=2 on the coarsest grid. Thus the blocking factor is set to amr.blocking_factor_y = 2.

AMR-Wind information

==============================================================================
                AMR-Wind (https://github.com/exawind/amr-wind)

  AMR-Wind version :: 31ef113
  AMR-Wind Git SHA :: 31ef1137b00b304b62b84edaa5b819c0bf0b7436
  AMReX version    :: 24.07-29-g3cb7f1555494

  Exec. time       :: Fri Aug 23 11:38:44 2024
  Build time       :: Jul 31 2024 10:30:51
  C++ compiler     :: GNU 11.3.0

  MPI              :: ON    (Num. ranks = 31)
  GPU              :: OFF
  OpenMP           :: OFF

  Enabled third-party libraries: 
    NetCDF    4.7.4
    HYPRE     2.30.0

           This software is released under the BSD 3-clause license.           
 See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details. 
------------------------------------------------------------------------------

[marchdf]

Can you try setting Ny = 4 and the blocking factor to 4? I know it's not ideal but I think it would help me understand something.

[tonyinme]

Hi @marchdf , yes, I was able to reproduce the issue with those changes.

[mbkuhn]

Not sure if this will fix it, but it is likely related. Could you try it out with this change applied?
#1222