GPU memory requirements for large-particle jobs after AMReX update #769
Assignees
Labels
bug: affects latest release
Bug also exists in latest release version
bug
Something isn't working
component: third party
Changes in ImpactX that reflect a change in a third-party library
Milestone
Comments
Merged
ax3l
added
bug
|
@WeiqunZhang and @RemiLehe identified this as a regression caused by AMReX-Codes/amrex#4175 A current work-around is to manually set with GPU-aware MPI. Additional to the memory overhead this introduced, and using this work-around, the runtime that @cemitch99 observes is significantly higher than before. |
5 tasks
5 tasks
|
I believe this PR will fix this in AMReX: AMReX-Codes/amrex#4286 |
atmyers
added a commit
to AMReX-Codes/amrex
that referenced
this issue
Jan 9, 2025
For mesh data it is better to use a small, separate Arena for communication. But, we found that for particle communication this approach uses too much memory (see the ImpactX Issue here: BLAST-ImpactX/impactx#769) This restores the old default behavior (prior to PR #4175), but allows users to opt-in to using The_Comms_Arena if desired. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After updating AMReX dependencies (as described in PR#760), a 1B-particle ImpactX job can no longer run because it exceeds GPU memory requirements. To illustrate this, input and output are included below (excluding particle-based diagnostic output to save space). Three cases are included: 1) running the job using ImpactX before updating the AMReX dependencies, 2) running the job after updating the dependencies (as-is), and 3) running the job after updating the dependencies and manually setting
the_arena_init_size. Here 1) runs successfully, 2) fails immediately due to a memory error, and 3) starts successfully but eventually fails.Test1BParticles.zip
The text was updated successfully, but these errors were encountered: