[Build] expected a type, got ‘Eigen::PropagateNaN’

[axbycc-mark]

Describe the issue

Building on Ubuntu 22.04, tag v1.20.1, using command ./build.sh --config RelWithDebInfo --build_shared_lib --parallel --compile_no_warning_as_error --skip_submodule_sync --use_cuda --cuda_home /usr/local/cuda-12.5 --cudnn_home /usr/lib/x86_64-linux-gnu.

The error was the following.

/home/axby/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: error: type/value mismatch at argument 1 in template parameter list for ‘template<class OtherDerived> const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<typename Eigen::internal::traits<T>::Scalar, typename Eigen::internal::traits<OtherDerived>::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase<Derived>::max(const Eigen::ArrayBase<OtherDerived>&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
  859 |               per_iter_bh.EigenInput0<T>().array().template max<Eigen::PropagateNaN>(
      |               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
  860 |                   per_iter_bh.EigenInput1<T>().array());
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~          
/home/axby/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: note:   expected a type, got ‘Eigen::PropagateNaN’

I did a string search of the whole repo for "PropagateNaN" and did not find any definitions for this expression. I understand it should be defined by the Eigen library, but I'm also not finding where Eigen is located in the source tree.

I found a related issue here #22679, but the user fixed it by switching to a different version.

Urgency

No response

Target platform

x86-64

Build script

./build.sh --config RelWithDebInfo --build_shared_lib --parallel --compile_no_warning_as_error --skip_submodule_sync --use_cuda --cuda_home /usr/local/cuda-12.5 --cudnn_home /usr/lib/x86_64-linux-gnu

Error / output

/home/axby/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: error: type/value mismatch at argument 1 in template parameter list for ‘template<class OtherDerived> const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<typename Eigen::internal::traits<T>::Scalar, typename Eigen::internal::traits<OtherDerived>::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase<Derived>::max(const Eigen::ArrayBase<OtherDerived>&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
  859 |               per_iter_bh.EigenInput0<T>().array().template max<Eigen::PropagateNaN>(
      |               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
  860 |                   per_iter_bh.EigenInput1<T>().array());
      |                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~          
/home/axby/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: note:   expected a type, got ‘Eigen::PropagateNaN’

Visual Studio Version

No response

GCC / Compiler Version

g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

[axbycc-mark]

This looks like a genuine issue with the onnxruntime_external_deps.cmake file. On line 545 I added some debug statements.

find_package(Eigen3 CONFIG)
if(Eigen3_FOUND)
  get_target_property(eigen_INCLUDE_DIRS Eigen3::Eigen INTERFACE_INCLUDE_DIRECTORIES)
  message(FATAL_ERROR "Eigen found: ${eigen_INCLUDE_DIRS}")
else()
  include(eigen) # FetchContent
  message(FATAL_ERROR "Eigen not found")
endif()

Re-running the build triggered the following message.

CMake Error at external/onnxruntime_external_deps.cmake:549 (message):
  Eigen found: /usr/include/eigen3
Call Stack (most recent call first):
  CMakeLists.txt:614 (include)

So cmake is trying to use my system eigen, even though I have not specified the --use_preinstalled_eigen flag.

[axbycc-mark]

Here is a patch that fixes the error.

diff --git a/cmake/external/onnxruntime_external_deps.cmake b/cmake/external/onnxruntime_external_deps.cmake
index 85746027d4..384282cab0 100644
--- a/cmake/external/onnxruntime_external_deps.cmake
+++ b/cmake/external/onnxruntime_external_deps.cmake
@@ -543,12 +543,7 @@ if(TARGET ONNX::onnx_proto AND NOT TARGET onnx_proto)
   add_library(onnx_proto ALIAS ONNX::onnx_proto)
 endif()
 
-find_package(Eigen3 CONFIG)
-if(Eigen3_FOUND)
-  get_target_property(eigen_INCLUDE_DIRS Eigen3::Eigen INTERFACE_INCLUDE_DIRECTORIES)
-else()
-  include(eigen) # FetchContent
-endif()
+include(eigen) # FetchContent
 
 if(onnxruntime_USE_VCPKG)
   find_package(wil CONFIG REQUIRED)

[eKevinHoang]

Look at this: #23413 (comment)

[axbycc-mark]

@eKevinHoang It looks to be the same bug. The issue is that the build script takes an argument called --use_preinstalled_eigen, but it is never checked, and the build always uses system Eigen, no matter what the flag is. This is because the flag is checked in eigen.cmake but that file is never looked at if a system Eigen is found.

[eKevinHoang]

@axbycc-mark You need to download Eigen from the source code mentioned in the comment above to build ONNX Runtime. The reason is that Eigen has not released a new version, and it is currently missing some required updates.

Installing Eigen is straightforward and supports make install.

Hope this helps!

[axbycc-mark]

@eKevinHoang You may be misunderstanding the situation. The build script is set up to automatically download the correct version of Eigen. It's supposed to happen as long as the user does not specify --use_preinstalled_eigen. But there is a logic error when checking the flag. The flag is always ignored when the user has a system eigen.

[snnn]

Please use the latest code from the main branch and set FETCHCONTENT_TRY_FIND_PACKAGE_MODE to NEVER if you do not want the build to prefer using system libs.

[axbycc-mark]

Thanks for the tip @snnn . However I think this is still a bug on the main branch. The flag below has no effect.

onnxruntime/tools/ci_build/build.py

Line 584 in 25f4274

parser.add_argument("--use_preinstalled_eigen", action="store_true", help="Use pre-installed Eigen.")

Whether you specify it or not, the build will use the system Eigen if it exists.

[snnn]

The "use_preinstalled_eigen" flag is deprecated. It was added for one special case.

[snnn]

If you have a preinstalled eigen, you can configure it in a way that cmake's find_package function can find it. Then Onnx runtime will consume it through find_package.