How to build openEMS on macOS?

[biergaizi]

I'm trying to collect performance data of openEMS on various CPUs, including Apple M1. Unfortunately, building openEMS on macOS is much harder than I expected, unlike what's suggested by the current documentation, and I hit several roadblocks.

The official documentation says to run,

brew tap thliebig/openems https://github.com/thliebig/openEMS-Project.git
brew install --HEAD openems

However, this has several problems:

1. Python extension is not enabled by default, but there's no information on how to change build options.

2. I need to build openEMS with custom code change for development, but there's no information on how to build openEMS outside Homebrew's packaging, or information about changing openEMS's code within the Homebrew package.

3. Trying to build openEMS manually outside Homebrew causes a fparser dynamic library path issue, which must be manually edited via install_name_tool, openEMS runs okay after changing fparser's search path. The CMake file probably needs a fix.

   • Update: I've identified and patched the bug (fparser's install name is not set correctly on macOS fparser#4), now it has been fixed.

4. Enabling Python module in the build works, but at runtime, it crashes at assert os.getcwd() == sim_path for some reason.

   • Update: I've identified and patched the bug (openEMS.pyx AssertionError due to checking non-canonical path name openEMS#113), now it has been fixed.

5. LLVM/clang cannot generate efficient vectorized code for the SSE/SIMD engine due to an unknown technical problem (this is not a macOS issue, but a LLVM/clang issue, since the problem exists on FreeBSD as well), so we must build openEMS with GCC to get a reasonable comparison. Unfortunately, using CC="gcc-12" and CXX="g++-12", CSXCAD fails to build due to a linking issue on macOS.

   • Update: As a workaround, you can use Gentoo Prefix or nixOS, both provide a GCC environment withit macOS.

I consider myself an experienced Linux/Unix sysadmin but I'm not familiar with macOS's development environment. If you have experience on using openEMS on macOS, please take a look at this (Update: after a few days of experimentation, including fixing several bugs, I've now gained a little bit of experience).

[biergaizi]

CC @aWZHY0yQH81uOYvH
If you have time, please take a look at this problem to offer some guidance, thanks.

[aWZHY0yQH81uOYvH]

The brew thing has just been figured out after patching a few things in openEMS that made it work with the newer versions of libraries brew installs on macOS. It's still missing all the build options and, as you say, the python bindings don't work even though they will install (if you do it manually). I haven't messed with it since because it was at a reasonably working state and school was starting up again.

My best attempt at compiling the entire package without Homebrew's magic CMake options was here. You can probably swap VTK back to the normal version brew installs instead of 8.2 and not bother with the unlinking steps since that should have been fixed in recent PRs. Basically, you need to make sure it uses C++17 and has the proper paths for everything the first time you run CMake. You then may need to modify DYLD_LIBRARY_PATH in the openEMS.sh script to make sure it can find its own libraries if you're not installing in a normal path. The top-level CMake file (in openEMS-Project) will not rebuild changes to the individual packages, so if you're experimenting with things it's a good idea to nuke your build folder between things to make sure your changes apply.

If you are just trying to benchmark things, it's much easier to compile just openEMS without the GUI stuff, and then you can simply re-use the same xml file across all platforms (and run straight from the command line without any Octave or Python stuff).

Regarding GCC vs clang, I did try quite a bit to get GCC to compile it, and also ran into lots of linking errors. I pretty much stopped once clang worked, so I don't really know how to proceed on that front. Macs are also increasingly going to ARM CPUs (and openEMS will compile and work on them), so would the x86-specific instructions and optimizations even be relevant? Unless you're trying to find differences between operating systems on the exact same hardware...

[biergaizi]

Macs are also increasingly going to ARM CPUs (and openEMS will compile and work on them), so would the x86-specific instructions and optimizations even be relevant?

The so-called "SSE engine" in openEMS, for the most part, actually does not use SSE directly. It uses GCC's vector extension to perform cross-platform semi-auto vectorization, based on the type annotation provided by the user. The implementation works like this: in tools/array_ops.h, it defines a special 128-bit vector data type named v4sf by deriving it from the float type, using GCC's vector_size annotation.

typedef float v4sf __attribute__ ((vector_size (F4VECTOR_SIZE))); // vector of four single floats
union f4vector
{
	v4sf v;
	float f[4];
};

Then the new vector data types v4sf and f4vector are used to define the arrays for simulations. The main "SSE" FDTD kernel actually doesn't use any SSE code at all, it's automatically generated by GCC and LLVM/clang. So this "SSE" engine is 80% cross-platform, excluding a few explicit SSE function calls. I think it should be renamed to "SIMD engine". On ARM, the compiler should generate NEON instructions on ARM, and on PowerPC we automatically have AltVec instructions.

The quality of the SIMD code obviously depends on the compiler backends. It's why I think it's extremely important to test both GCC and clang. On x86_64, clang's vectorized code has a speed that is only a fraction as fast as GCC for reasons that I still need to investigate. A similar situation on other platforms cannot be ruled out.

[sayalok]

I have to inject openEMS inside Python (Django) and run this using docker. It's much more easier and flexible to run openEMS in the docker. So for now you could try that

[JerryMartinsonLeviti]

This sounds like a great idea. Do you know of a ready-made docker image that has everything setup and configured right?

[biergaizi]

I was finally able to run openEMS on macOS in a fully portable manner, including Python and GCC. My solution was Gentoo Prefix. Gentoo Prefix works by installing the entire Gentoo userland into an unprivileged directory, similar to how a user can install software under /home. This way, one can use pre-existing Gentoo packages and toolchains on a foreign Unix system without changing the base system.

I only managed to do this after fixing multiple portability bugs in the toolchains and system libraries, since Apple Silicon was a previously untested target for Gentoo... As a 10-year Gentoo user, when you have a hammer, everything looks like a nail...

But now it works. I should be able to publish some performance data about openEMS on Apple Silicon soon.

[biergaizi]

Enabling Python module in the build works, but at runtime, it crashes at assert os.getcwd() == sim_path for some reason.

Upon further inspection, this is in fact Bug thliebig/openEMS#113. I've submitted a patch (thliebig/openEMS#114) for solving this problem.

[biergaizi]

Trying to build openEMS manually outside Homebrew causes a fparser dynamic library path issue, which must be manually edited via install_name_tool, openEMS runs okay after changing fparser's search path. The CMake file probably needs a fix.

I just diagnosed this problem. Right now, fparser's CMake script doesn't set its "install name" as a full install path on macOS, unlike other libraries in openEMS (which all did this correctly):

gentoo@users-iMac-Pro ~/code/openEMS-Project/CSXCAD/build $ otool -D ~/opt/lib/libCSXCAD.dylib
/Users/gentoo/opt/lib/libCSXCAD.dylib:
/Users/gentoo/opt/lib/libCSXCAD.0.dylib

gentoo@users-iMac-Pro ~/code/openEMS-Project/CSXCAD/build $ otool -D ~/opt/lib/libfparser.4.5.1.dylib
/Users/gentoo/opt/lib/libfparser.4.5.1.dylib:
libfparser.4.dylib

This is why only libfparser cannot be found, but other shared libraries can. After CSXCAD and openEMS are built, this incomplete install name will propagate upwards, so all binaries will have a wrong search path.

The solution is adding two lines of code in CMakeLists.txt:

if(APPLE AND NOT CMAKE_INSTALL_NAME_DIR)
  set(CMAKE_INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/lib${LIB_SUFFIX}")
endif()

After this change, the "install name" of fparser is now corrected:

gentoo@users-iMac-Pro ~/code/openEMS-Project/fparser $ otool -D ~/opt/lib/libfparser.4.5.1.dylib 
/Users/gentoo/opt/lib/libfparser.4.5.1.dylib:
/Users/gentoo/opt/lib/libfparser.4.dylib

I'm now working on a Pull Request.

Update: problem reported and pull request submitted.

[biergaizi]

Patch has been accepted, this problem is fixed.

[biergaizi]

In case anyone's wondering, the updated macOS installation instructions have already been included in the online documentation.

• Homebrew installation (with Python)
  https://docs.openems.de/install/package.html#macos

• Source installation
  https://docs.openems.de/install/requirements.html#macos
  https://docs.openems.de/install/manual.html

The only thing that hasn't been covered is building openEMS with GCC instead of clang on macOS. But if you're doing this, you probably don't need further guidance (due to potential LLVM/GCC conflict, I recommend using nixOS or Gentoo Prefix to start with an all-GNU toolchain). If I recall correctly, GCC and clang builds for ARM64 macOS have similar performance, so there's no need to build openEMS with GCC.