[Build] Cross build failure for CPU architecture ARM Cortex A53 with Flags "-march=armv8-a -mtune=cortex-a53" due to Eigen3 library.

[shadab205]

Describe the issue

I am trying to build onnxruntime for a Raspberry Pi 3 with the CPU core architecture Cortex-A53 but the build fails and the issue are pointing to Eigen3 Library.

I am cross building it on a WSL Ubuntu 22.04 setup on a Windows machine. But I also tried to build it natively on the Raspberry Pi with similar failures.

Urgency

The build failure is blocking a project from moving forward.

Target platform

Linux aarch64

Build script

cmake .. -Donnxruntime_ENABLE_CPUINFO=OFF -Donnxruntime_GCC_STATIC_CPP_RUNTIME=ON -DCMAKE_C_FLAGS="-march=armv8-a -mtune=cortex-a53 -Wno-error" -DCMAKE_CXX_FLAGS="-march=armv8-a -mtune=cortex-a53 -Wno-error" Donnxruntime_DEV_MODE=OFF -DCMAKE_TOOLCHAIN_FILE=arm64.toolchain.cmake -DCMAKE_BUILD_TYPE=Release -Donnxruntime_BUILD_UNIT_TESTS=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_USE_CUDA=OFF -Donnxruntime_USE_DML=OFF

---

Custom Toolchain File:

Specify the cross compiler

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

Specify the cross compiler

set(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
set(CMAKE_FIND_ROOT_PATH /usr/aarch64-linux-gnu)

Search for programs in the build host directories

set(CMAKE_FIND_PROGRAMS)

Search for headers and libraries in the target environment

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

Error / output

/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:855:85: error: type/value mismatch at argument 1 in template parameter list for ‘template const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<typename Eigen::internal::traits::Scalar, typename Eigen::internal::traits::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase::max(const Eigen::ArrayBase&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
855 | per_iter_bh.EigenInput0().array().template maxEigen::PropagateNaN(per_iter_bh.ScalarInput1());
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:855:85: note: expected a type, got ‘Eigen::PropagateNaN’
/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: error: no matching function for call to ‘Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >::maxEigen::PropagateNaN(Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >)’
859 | per_iter_bh.EigenInput0().array().template maxEigen::PropagateNaN(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
860 | per_iter_bh.EigenInput1().array());
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/eigen3/Eigen/Core:19,
from /home/user/onnxruntime/onnxruntime/core/util/math_cpuonly.h:68,
from /home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.h:10,
from /home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:4:
/usr/include/eigen3/Eigen/src/Core/../plugins/ArrayCwiseBinaryOps.h:59:28: note: candidate: ‘template const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<typename Eigen::internal::traits::Scalar, typename Eigen::internal::traits::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase::max(const Eigen::ArrayBase&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
59 | EIGEN_MAKE_CWISE_BINARY_OP(max,max)
| ^~~
/usr/include/eigen3/Eigen/src/Core/util/Macros.h:1339:4: note: in definition of macro ‘EIGEN_MAKE_CWISE_BINARY_OP’
1339 | (METHOD)(const EIGEN_CURRENT_STORAGE_BASE_CLASS &other) const
| ^~~~~~
/usr/include/eigen3/Eigen/src/Core/../plugins/ArrayCwiseBinaryOps.h:59:28: note: template argument deduction/substitution failed:
59 | EIGEN_MAKE_CWISE_BINARY_OP(max,max)
| ^~~
/usr/include/eigen3/Eigen/src/Core/util/Macros.h:1339:4: note: in definition of macro ‘EIGEN_MAKE_CWISE_BINARY_OP’
1339 | (METHOD)(const EIGEN_CURRENT_STORAGE_BASE_CLASS &other) const
| ^~~~~~
/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: error: type/value mismatch at argument 1 in template parameter list for ‘template const Eigen::CwiseBinaryOp<Eigen::internal::scalar_max_op<typename Eigen::internal::traits::Scalar, typename Eigen::internal::traits::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase::max(const Eigen::ArrayBase&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<long unsigned int, -1, 1, 0, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
859 | per_iter_bh.EigenInput0().array().template maxEigen::PropagateNaN(
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
860 | per_iter_bh.EigenInput1().array());
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:859:85: note: expected a type, got ‘Eigen::PropagateNaN’

Visual Studio Version

No response

GCC / Compiler Version

aarch64-linux-gnu v11.4.0

[vlasov01]

I have the same error as well. But I'm building it directly on the ARM Cortex-A9 device (Linux pynq 5.15.19-xilinx-v2022.1 #1 SMP PREEMPT Mon Apr 11 17:52:14 UTC 2022 armv7l armv7l armv7l GNU/Linux ) using cmake version 3.31.20241016-gee2eadb.

Here is my build command:
./build.sh --config Release --parallel --skip_submodule_sync --skip_tests --use_xnnpack --build_wheel --cmake_extra_defines CMAKE_CXX_FLAGS="-Wno-dev -Wall -Wextra -O3 -g3 -mfloat-abi=hard -mfpu=neon-vfpv3 -mcpu=cortex-a9 -mfloat-abi=hard" CMAKE_C_FLAGS="-Wno-dev -Wall -Wextra -O3 -g3 -mfloat-abi=hard -mfpu=neon-vfpv3 -mcpu=cortex-a9 -mfloat-abi=hard"

[sevagh]

I'm having the same issue building for Linux x86_64 on an x86_64 computer:

/home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc: In member function ‘onnxruntime::common::Status onnxruntime::Min_6<T>::Compute(onnxruntime::OpKernelContext*) const [with T = float]’:
/home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:708:56: error: no matching function for call to ‘Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1>, 0, Eigen::Stride<0, 0> > >::min<Eigen::PropagateNaN>(Eigen::ArrayWrapper<Eigen::Map<const Eigen::Matrix<float, -1, 1>, 0, Eigen::Stride<0, 0> > >)’
  708 |     min = min.array().template min<Eigen::PropagateNaN>(EigenMap<float>(data_n).array());
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/eigen3/Eigen/Core:19,
                 from /home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/util/math_cpuonly.h:68,
                 from /home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.h:10,
                 from /home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:4:
/usr/include/eigen3/Eigen/src/Core/../plugins/ArrayCwiseBinaryOps.h:33:28: note: candidate: ‘template<class OtherDerived> const Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<typename Eigen::internal::traits<T>::Scalar, typename Eigen::internal::traits<OtherDerived>::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase<Derived>::min(const Eigen::ArrayBase<OtherDerived>&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
   33 | EIGEN_MAKE_CWISE_BINARY_OP(min,min)
      |                            ^~~
/usr/include/eigen3/Eigen/src/Core/util/Macros.h:1339:4: note: in definition of macro ‘EIGEN_MAKE_CWISE_BINARY_OP’
 1339 |   (METHOD)(const EIGEN_CURRENT_STORAGE_BASE_CLASS<OtherDerived> &other) const \
      |    ^~~~~~
/usr/include/eigen3/Eigen/src/Core/../plugins/ArrayCwiseBinaryOps.h:33:28: note:   template argument deduction/substitution failed:
   33 | EIGEN_MAKE_CWISE_BINARY_OP(min,min)
      |                            ^~~
/usr/include/eigen3/Eigen/src/Core/util/Macros.h:1339:4: note: in definition of macro ‘EIGEN_MAKE_CWISE_BINARY_OP’
 1339 |   (METHOD)(const EIGEN_CURRENT_STORAGE_BASE_CLASS<OtherDerived> &other) const \
      |    ^~~~~~
/home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:708:56: error: type/value mismatch at argument 1 in template parameter list for ‘template<class OtherDerived> const Eigen::CwiseBinaryOp<Eigen::internal::scalar_min_op<typename Eigen::internal::traits<T>::Scalar, typename Eigen::internal::traits<OtherDerived>::Scalar>, const Derived, const OtherDerived> Eigen::ArrayBase<Derived>::min(const Eigen::ArrayBase<OtherDerived>&) const [with OtherDerived = OtherDerived; Derived = Eigen::ArrayWrapper<Eigen::Map<Eigen::Matrix<float, -1, 1>, 0, Eigen::Stride<0, 0> > >]’
  708 |     min = min.array().template min<Eigen::PropagateNaN>(EigenMap<float>(data_n).array());
      |           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/sevagh/repos/basicpitch.cpp/vendor/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.cc:708:56: note:   expected a type, got ‘Eigen::PropagateNaN’

[sevagh]

What version of Eigen are you using? I was able to bypass this issue by pointing Onnxruntime to use a local copy of Eigen which is newer than my system Eigen3.

The 3.4 branch doesn't have it, you need to use Eigen from master.

From your original log:
/usr/include/eigen3/Eigen/

Seems to me you installed Eigen from your system package manager, which is the same as mine (and those are generally oudated).

However, pointing to your own custom path of Eigen with the build options is a bit tricky: #22515 - it will default to your system's Eigen even if you supply --eigen_path=/path/to/newer/eigen. I commented out the find_package call to force it to obey my supplied eigen_path:

--- a/cmake/external/onnxruntime_external_deps.cmake
+++ b/cmake/external/onnxruntime_external_deps.cmake
@@ -543,12 +543,12 @@ if(TARGET ONNX::onnx_proto AND NOT TARGET onnx_proto)
   add_library(onnx_proto ALIAS ONNX::onnx_proto)
 endif()

-find_package(Eigen3 CONFIG)
-if(Eigen3_FOUND)
-  get_target_property(eigen_INCLUDE_DIRS Eigen3::Eigen INTERFACE_INCLUDE_DIRECTORIES)
-else()
-  include(eigen) # FetchContent
-endif()
+#find_package(Eigen3 CONFIG)
+#if(Eigen3_FOUND)
+#  get_target_property(eigen_INCLUDE_DIRS Eigen3::Eigen INTERFACE_INCLUDE_DIRECTORIES)
+#else()
+include(eigen) # FetchContent
+#endif()

[vlasov01]

It seems I was able to bypass this issue by removing my system Eigen3. I'm not 100% sure as the build is still in progress (91%).

sudo apt remove libeigen3-dev

[sevagh]

Makes sense to me - if you delete your outdated system Eigen3, onnxruntime will download a newer version which supports this template

[ep1cman]

I encountered this issues too. It turns out the cmake scripts are a bit flawed. If egien is found on the system it will always use that. It will never import the cmake file and thus it will not honour the --use_preinstalled_eigen flag not being set. Not really sure what the intended purpose of this flag is but currently, if there is a system provided eigen, it will always use that.

To get the build work for me I just commented out some lines in the cmake such that it always runs the eigen cmake file and downloads its own copy.

[sevagh]

I encountered this issues too. It turns out the cmake scripts are a bit flawed. If egien is found on the system it will always use that. It will never import the cmake file and thus it will not honour the --use_preinstalled_eigen flag not being set. Not really sure what the intended purpose of this flag is but currently, if there is a system provided eigen, it will always use that.

To get the build work for me I just commented out some lines in the cmake such that it always runs the eigen cmake file and downloads its own copy.

FYI, I made an issue for this: #22515

[vlasov01]

Makes sense to me - if you delete your outdated system Eigen3, onnxruntime will download a newer version which supports this template

Actually, I had the latest release of libeigen3-dev 3.4.0-2ubuntu2.
As I understand onnxruntime is using a release candidate of it as per #18286.

[shadab205]

Makes sense to me - if you delete your outdated system Eigen3, onnxruntime will download a newer version which supports this template

Actually, I had the latest release of libeigen3-dev 3.4.0-2ubuntu2. As I understand onnxruntime is using a release candidate of it as per #18286.

I have 3.4.0-2ubuntu2 version installed as well. I will try to remove the system package and try again.

[shadab205]

I encountered this issues too. It turns out the cmake scripts are a bit flawed. If egien is found on the system it will always use that. It will never import the cmake file and thus it will not honour the --use_preinstalled_eigen flag not being set. Not really sure what the intended purpose of this flag is but currently, if there is a system provided eigen, it will always use that.

To get the build work for me I just commented out some lines in the cmake such that it always runs the eigen cmake file and downloads its own copy.

Sebastian Goscik (@ep1cman) Can you tell us what changes did you do to the cmake files? Is it the same as suggested by Sevag H (@sevagh) above

[snnn]

Does it latest eigen version work? You may open cmake/deps.txt then search the eigen line and update it. The last column is the SHA1 sum of the source zip file.

If it works. I will submit a PR to update it.

[shadab205]

I will try to compile it again.

[shadab205]

Does it latest eigen version work? You may open cmake/deps.txt then search the eigen line and update it. The last column is the SHA1 sum of the source zip file.

If it works. I will submit a PR to update it.

Changming Sun (@snnn) I can confirm that the issue is resolved( and as a plus the issue with the nsync has been resolved). I will close this issue. Thanks.