openPMD Beam Input via `Source` Element #820

ax3l · 2025-01-30T05:03:49Z

Add a new element Source that reads openPMD files from our beam monitors. Add an example in Python. Close #97

This can also be used to restart a simulation that was checkpointed at a beam monitor.

Python example
input file example
documentation

Follow-Up Ideas

This is designed intentionally as a new element. In the future, we probably restructure our initial distribution logic to also go into this element, so that "from file" is just one of its options.

Not needed for Sirepo, but we will also add an option to use this from inputs files:

For other projects, we will implement t-to-s transforms (for WarpX) and other options, as documented in inline TODOs.
We will add a control option: restore reference particle (for restarts) yes/no

src/particles/elements/Source.cpp

examples/fodo_channel_restart/run_fodo.py

examples/fodo_channel_restart/README.rst

examples/fodo_channel_restart/input_fodo.in

src/python/ImpactXParticleContainer.cpp

src/initialization/InitDistribution.cpp

src/initialization/InitParser.cpp

ax3l · 2025-02-08T00:54:28Z

Ok I got all functionality implemented, but will pick another example.

The FODO channel demos another use case with the skipping of beam monitor outputs, which becomes to overloaded and involved to base the restart demo on...

Update: done.

ax3l · 2025-02-10T23:32:26Z

examples/solenoid_restart/analysis_solenoid.py

+print(f"  rtol={rtol} (ignored: atol~={atol})")
+
+assert np.allclose(
+    [sigx, sigy, emittance_x, emittance_y, emittance_t],


I skipped the sigt comparison here because I did not know the final value after a total of 4 solenoids and there seems to be some dispersion.

From Slack:

Here sigma_t should evolve just as it would in a drift space. It is not matched, since there is no longitudinal focusing. So it should be ignored if you’re looking at how well the initial and final distributions agree.

src/elements/diagnostics/BeamMonitor.cpp

+        /*
+        using SrcData = WarpXParticleContainer::ParticleTileType::ConstParticleTileDataType;
+        tmp.copyParticles(*pc,
+                          [=] AMREX_GPU_HOST_DEVICE (const SrcData& src, int ip, const amrex::RandomEngine& engine)
+                          {
+                              const SuperParticleType& p = src.getSuperParticle(ip);
+                              return random_filter(p, engine) * uniform_filter(p, engine)
+                                     * parser_filter(p, engine) * geometry_filter(p, engine);
+                          }, true);
+        */


src/elements/diagnostics/BeamMonitor.cpp

+            /*
+            // comment this in once IntSoA::nattribs is > 0
+
+            std::copy(IntSoA::names_s.begin(), IntSoA::names_s.end(), int_soa_names.begin());
+
+            for (auto int_idx=0; int_idx < RealSoA::nattribs; int_idx++) {
+                auto const component_name = int_soa_names.at(int_idx);
+                getComponentRecord(component_name).storeChunkRaw(
+                    soa.GetIntData(int_idx).data(), {offset}, {numParticleOnTile64});
+            }
+            */


cemitch99

The logic looks good to me. The app benchmark works for me. However, I was unable to run the Python test successfully. We might need to check the updates to elements.cpp.

ax3l · 2025-02-11T02:35:19Z

What kind of error did you get in the Python example?

Did you try via

ctest --test-dir build --output-on-failure -R solenoid

?

cemitch99 · 2025-02-11T02:53:23Z

Initializing AMReX (25.01-29-g69f1ac884c6a)...
OMP initialized with 12 OMP threads
AMReX (25.01-29-g69f1ac884c6a) initialized

Grids Summary:
  Level 0   1 grids  512 cells  100 % of domain

[AbstractIOHandlerImpl] IO Task OPEN_FILE failed with exception. Clearing IO queue and passing on the exception.
Traceback (most recent call last):
  File "/Users/cemitch/src/impactx/examples/solenoid_restart/run_solenoid.py", line 33, in <module>
    push(pc, elements.Source("openPMD", "../solenoid.py/diags/openPMD/monitor.h5"))
RuntimeError: Read Error in backend HDF5
Object type:	File
Error type:	Inaccessible
Further description:	Supplied directory is not valid: ../solenoid.py/diags/openPMD/

cemitch99 · 2025-02-11T02:55:46Z

The problem does not occur using ctest. The error above occurs when running run_solenoid.py manually within examples/. It looks like the path in Line 33 of run_solenoid.py is the issue.

ax3l · 2025-02-11T03:08:57Z

Ah perfect, the error just says it cannot find the file to load under the provided path:

File "/Users/cemitch/src/impactx/examples/solenoid_restart/run_solenoid.py", line 33, in
push(pc, elements.Source("openPMD", "../solenoid.py/diags/openPMD/monitor.h5"))
...
Further description: Supplied directory is not valid: ../solenoid.py/diags/openPMD/

Instead of using ../solenoid.py/diags/openPMD/monitor.h5 you could run the solenoid example first (either app or Python), then use the path file to its diags/openPMD/monitor.h5 file.

I think what happened is that you ran the app version of the regular solenoid example, then the two restart files. That works, but you need to replace ../solenoid.py/ with ../solenoid/ in the path of the Python restart test then. Currently, I made the Python script depend on the Python solenoid data and the app restart input base on the app data.

examples/solenoid_restart/README.rst

cemitch99

Looks good.

ax3l added component: elements Elements/external fields component: distributions beam distributions labels Jan 30, 2025

ax3l added this to the Sirepo (FY24/24 Phase 1 SBIR) milestone Jan 30, 2025

ax3l requested a review from cemitch99 January 30, 2025 05:04

ax3l commented Jan 30, 2025

View reviewed changes

src/particles/elements/Source.cpp Outdated Show resolved Hide resolved

ax3l commented Jan 30, 2025

View reviewed changes

examples/fodo_channel_restart/run_fodo.py Outdated Show resolved Hide resolved

cemitch99 reviewed Jan 30, 2025

View reviewed changes

examples/fodo_channel_restart/README.rst Outdated Show resolved Hide resolved

cemitch99 reviewed Jan 30, 2025

View reviewed changes

examples/fodo_channel_restart/input_fodo.in Outdated Show resolved Hide resolved

cemitch99 reviewed Jan 30, 2025

View reviewed changes

src/python/ImpactXParticleContainer.cpp Outdated Show resolved Hide resolved

ax3l force-pushed the topic-read-openpmd branch from 01ccc1e to 160e870 Compare February 7, 2025 07:30

ax3l commented Feb 7, 2025

View reviewed changes

src/initialization/InitDistribution.cpp Outdated Show resolved Hide resolved

ax3l commented Feb 7, 2025

View reviewed changes

src/initialization/InitParser.cpp Outdated Show resolved Hide resolved

ax3l force-pushed the topic-read-openpmd branch from 160e870 to 5134da6 Compare February 8, 2025 00:51

ax3l force-pushed the topic-read-openpmd branch 2 times, most recently from 2c731fc to f87f83f Compare February 10, 2025 23:31

ax3l commented Feb 10, 2025

View reviewed changes

ax3l assigned cemitch99 Feb 10, 2025

ax3l force-pushed the topic-read-openpmd branch 2 times, most recently from 487662d to 1f763ad Compare February 11, 2025 00:30

ax3l added 4 commits February 10, 2025 16:34

Source Element for openPMD

bf382ea

Rename Source Files of BeamMonitor

8c8eb94

Deduplicate openPMD Logic Helpers

a215f2d

Docs

f0bb85a

ax3l force-pushed the topic-read-openpmd branch from 1f763ad to d2d99fe Compare February 11, 2025 00:35

Example

f03a11e

ax3l force-pushed the topic-read-openpmd branch from d2d99fe to f03a11e Compare February 11, 2025 01:31

github-advanced-security bot found potential problems Feb 11, 2025

View reviewed changes

cemitch99 reviewed Feb 11, 2025

View reviewed changes

ax3l commented Feb 11, 2025

View reviewed changes

examples/solenoid_restart/README.rst Outdated Show resolved Hide resolved

Doc: Fix Copy-Paste

eabe2c3

cemitch99 approved these changes Feb 11, 2025

View reviewed changes

ax3l merged commit 8f03503 into BLAST-ImpactX:development Feb 11, 2025
16 checks passed

ax3l deleted the topic-read-openpmd branch February 11, 2025 18:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

openPMD Beam Input via `Source` Element #820

openPMD Beam Input via `Source` Element #820

ax3l commented Jan 30, 2025 •

edited

Loading

ax3l commented Feb 8, 2025 •

edited

Loading

ax3l Feb 10, 2025

ax3l Feb 11, 2025

cemitch99 left a comment

ax3l commented Feb 11, 2025

cemitch99 commented Feb 11, 2025 •

edited by ax3l

Loading

cemitch99 commented Feb 11, 2025 •

edited

Loading

ax3l commented Feb 11, 2025 •

edited

Loading

cemitch99 left a comment

openPMD Beam Input via Source Element #820

openPMD Beam Input via Source Element #820

Conversation

ax3l commented Jan 30, 2025 • edited Loading

Follow-Up Ideas

ax3l commented Feb 8, 2025 • edited Loading

ax3l Feb 10, 2025

Choose a reason for hiding this comment

ax3l Feb 11, 2025

Choose a reason for hiding this comment

cemitch99 left a comment

Choose a reason for hiding this comment

ax3l commented Feb 11, 2025

cemitch99 commented Feb 11, 2025 • edited by ax3l Loading

cemitch99 commented Feb 11, 2025 • edited Loading

ax3l commented Feb 11, 2025 • edited Loading

cemitch99 left a comment

Choose a reason for hiding this comment

openPMD Beam Input via `Source` Element #820

openPMD Beam Input via `Source` Element #820

ax3l commented Jan 30, 2025 •

edited

Loading

ax3l commented Feb 8, 2025 •

edited

Loading

cemitch99 commented Feb 11, 2025 •

edited by ax3l

Loading

cemitch99 commented Feb 11, 2025 •

edited

Loading

ax3l commented Feb 11, 2025 •

edited

Loading