ROCm workaround: Use ParallelFor instead of Reduce

[WeiqunZhang]

Assuming the failure is not often, we can use ParallelFor with atomicAdd to
obtain the number of failures. With this change, the ROCm memory issue seems
to be gone.

[zingale]

this fixes #2569

[BenWibking]

@WeiqunZhang would you recommend that we use atomics instead of amrex::ParReduce in Quokka as well:
https://github.com/quokka-astro/quokka/blob/2d863d655ec1e1f684f55ea745021b9c97a03c60/src/hydro_system.hpp#L204

(We also use amrex::ReduceToPlane:
https://github.com/quokka-astro/quokka/blob/2d863d655ec1e1f684f55ea745021b9c97a03c60/src/simulation.hpp#L2269)

[WeiqunZhang]

No, not for the functions in your links. Those kernels are small. Only if the kernel is very big and you see an issue, you might try to use atomics. Note that using atomics is very slow if it needs to be done a lot of times.

[BenWibking]

No, not for the functions in your links. Those kernels are small. Only if the kernel is very big and you see an issue, you might try to use atomics. Note that using atomics is very slow if it needs to be done a lot of times.

Is there a way to identify kernels that might be affected by the same issue?

The kernel that calls our reaction network is here: https://github.com/quokka-astro/quokka/blob/2d863d655ec1e1f684f55ea745021b9c97a03c60/src/Chemistry.hpp#L43

The reaction network itself (6k LOC, symbolically generated, hundreds of local variables) is here:
https://github.com/AMReX-Astro/Microphysics/blob/main/networks/primordial_chem/actual_rhs.H#L20

We see errors like this when running simulations with this network:

Memory access fault by GPU node-8 (Agent handle: 0x2975b60) on address 0x800033773000. Reason: Unknown.

[WeiqunZhang]

You can try to insert Gpu::streamSynchronize like in this PR. Without it, I saw

:0:rocdevice.cpp            :2724: 441955224434 us: [pid:105317 tid:0x7fffdd5ff700] Callback: Queue 0x7ffec2f00000 Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES: T
he runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. 
Code: 0x1008 Available Free mem : 13506 MB

It might help in your case.

[BenWibking]

You can try to insert Gpu::streamSynchronize like in this PR. Without it, I saw

:0:rocdevice.cpp            :2724: 441955224434 us: [pid:105317 tid:0x7fffdd5ff700] Callback: Queue 0x7ffec2f00000 Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES: T
he runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. 
Code: 0x1008 Available Free mem : 13506 MB

It might help in your case.

We can try that. You are referring to the call immediately after the burn kernel on line 417, right?

[WeiqunZhang]

Yes.

[WeiqunZhang]

It might also help to turn on tiling MFIter mfi(mf, IntVect(32,32,2048)) for the failed MFIter loop. If you do that, you need to use tilebox(), not validbox().

[BenWibking]

It might also help to turn on tiling MFIter mfi(mf, IntVect(32,32,2048)) for the failed MFIter loop. If you do that, you need to use tilebox(), not validbox().

We could try that. Does this change the kernel launch parameters (and/or, do you know why it helps)?

[WeiqunZhang]

It changes the number of blocks in the kernel launch.