Cannot open include file: 'caffe/include_symbols.hpp': No such file or directory

[lunzueta]

Hi @willyd. I've followed the new installation instructions (as explained here: https://github.com/willyd/caffe/tree/windows-cmake), but when compiling I get the error I've put in the title. I can't find that file anywhere in the repo nor in the cmake-generated build folder. It's only referenced in the file C:\caffe\build\caffe/export.hpp

What should be done?

Thanks

[willyd]

@lunzueta are you using the Visual Studio or Ninja generator? I mostly use Ninja, I tried to make sure that VS works too (see the build scripts in scripts/appveyor and also the build logs on appveyor).

What is your version of CMake? You need 3.4+. I use 3.5 and 3.6.

Are you building a shared library, i.e. BUILD_SHARED_LIBS=ON? If yes, I might have an idea on what's wrong.

In main CMakeLists.txt change:

# definition needed to include CMake generated files
add_definitions(-DCMAKE_WINDOWS_BUILD)

to

if(NOT BUILD_SHARED_LIBS)
  # definition needed to include CMake generated files
  add_definitions(-DCMAKE_WINDOWS_BUILD)
endif()

If that solves the problem can you submit a PR to the windows-cmake branch? I'll merge it.

[lunzueta]

Initially, I tried to use Ninja, but I couldn't configure the project properly with the cmake command line (I got some configuration errors). So, then I used cmake-gui, configured it selecting the generator "Visual Studio 12 2013 Win64", and for those dependencies that weren't directly detected by cmake, I manually included the paths to those downloaded by the conda package manager to C:\Anaconda2\envs\caffe\Library

I'm using CMake 3.5.1 and I had BUILD_SHARED_LIBS=OFF

[willyd]

If you can repro the CMake configure errors you got with Ninja and output that to a file I would be glad to look at it.

The caffe include_symbols.hpp is generated by some post_build event. For some reason it may not be executed at all or just fail to generate the file. You can put some debug message(STATUS "this is a debug message") in this file and see if anything shows up in the output window in VS. Put your debug message(STATUS "") in the if(CMAKE_SCRIPT_MODE_FILE).

[lunzueta]

I attach the two log files that generates CMake so that you can see what happened when I tried to configure it for Ninja. I think that the problem might be that even if I run the command "%VS120COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 to setup the msvc compiler, it strangely gets MinGW, and this might fail in later stages of the configuration.

CMakeOutput.txt
CMakeError.txt

[willyd]

Weird, maybe CMake checks for Mingw first? I don't have gcc installed but appveyor does and it does not conflict. Maybe this post can be of help?

Otherwise it should work from VS too. Make sure you start the VS devenv from the same command line you activated the conda environment.

If you can get the build output from VS and attach it here I will have a look.

[lunzueta]

Thanks @willyd, that helped. In the end, I used the following command to specify the compiler, and by the way to skip cudnn, because otherwise I had some compilation errors later:

cmake -GNinja -DCMAKE_C_COMPILER=cl.exe -DCMAKE_CXX_COMPILER=cl.exe -DUSE_CUDNN=OFF -DBLAS=Open -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DCMAKE_MODULE_PATH=%CMAKE_MODULE_PATH% -DCMAKE_PREFIX_PATH=%CMAKE_PREFIX_PATH% -DCMAKE_INSTALL_PREFIX=.\install ..\

So, I could compile Caffe with Ninja, and I was able to use the generated pycaffe with NVIDIA DIGITS in Windows 10 (I could train LeNet with MNIST correctly). But afterwards, I tried to use the C++ Caffe library externally and I had some problems generating the Visual Studio solution of the project with CMake (I don't want to use Ninja in this case, at least for now). I used find_package(Caffe) with the generated CaffeConfig.cmake file, and I had the errors I attach here with some of the dependencies. Any suggestions?

By the way, do you think it's necessary to include CMake in the caffe environment installed in Anaconda? For example, the cmake-gui installed there can't run, because there are some Qt files missing (I guess they'd be those added when running windeployqt.exe, included in Qt's bin folder). I'd rather use the CMake program I already had installed. Same for OpenCV.

cmake-output.txt

[willyd]

@lunzueta You need to pass the -DCMAKE_MODULE_PATH=%CMAKE_MODULE_PATH% -DCMAKE_PREFIX_PATH=%CMAKE_PREFIX_PATH% options to your CMake configure too. If you use cmake-gui you can set those variables too. Make sure you activate the caffe environment and call set_cmake_vars.bat.

You need to do so because the conda packages you installed (when following the README.md) provided custom FindBoost.cmake, FindZLIB.cmake, etc. CMAKE_MODULE_PATH defines where CMake looks to find those files. CMAKE_PREFIX_PATH defines where cmake looks for config.cmake files. This is necessary to ensure that the packages can be built on a machine and used on another, i.e. avoiding absolute include and library paths.

Yeah I got the same Qt error when trying to use the cmake-gui that comes with the cmake conda package. If I understand correctly this cmake-gui can be fixed by running windeployqt.exe. Where can I find this executable? You can definitely use your own cmake. You can either prepend to your path to make sure the system cmake is used or force the uninstall in your conda env using:

> conda uninstall cmake --force

[lunzueta]

Hi @willyd, I've done different tests to link Caffe externally to a project, but I still have this problem. I attach a couple of files: (1) a simple CMakeLists.txt file to generate a dummy library which links externally Caffe and (2) the results cmake gives me, when generating it with Ninja, like this (after preparing the conda environment):

cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=cl.exe -DCMAKE_CXX_COMPILER=cl.exe -DCMAKE_MODULE_PATH=%CMAKE_MODULE_PATH% -DCMAKE_PREFIX_PATH=%CMAKE_PREFIX_PATH% ..\

Thanks for your help

CMakeLists.txt
cmake-output.txt

[willyd]

Hi @lunzueta, the CaffeConfig.cmake will not resolve the other libs for you, this is something I still need to fix. Try with this (untested) CMakeLists.txt instead.
CMakeLists.txt

[willyd]

BTW, it should work with the Visual Studio generator too. Let me know if it does not.

[lunzueta]

OK, understood. I thought this could be done somehow from CaffeConfig.cmake, but I didn't check this possibility in detail. I forgot to remark that you can find the file windeployqt.exe in the bin folder of your Qt installation. This step is necessary for deploying the programs that have Qt as a dependence (you compiled cmake from source, did you?). Here you have some instructions of how to use it: http://doc.qt.io/qt-5/windows-deployment.html

[willyd]

I thought this could be done somehow from CaffeConfig.cmake, but I didn't check this possibility in detail.

You are right the CaffeConfig.cmake file should include calls to find_package(Boost) or rather to find_dependency and so on so that a consuming project like yours only has to do find_package(Caffe).

forgot to remark that you can find the file windeployqt.exe in the bin folder of your Qt installation. This step is necessary for deploying the programs that have Qt as a dependence (you compiled cmake from source, did you?). Here you have some instructions of how to use it: http://doc.qt.io/qt-5/windows-deployment.html

Thanks for the info. No I did not build cmake from source. I am just using the conda-forge cmake package. I reported the issue upstream.

[lunzueta]

Hi @willyd. I'm still a bit stuck with this. I used that CMake code you shared to find the dependencies, a little bit changed, like this, because LMDB, LevelDB and Snappy shouldn't appear as Boost components:

FIND_PACKAGE( Boost COMPONENTS atomic chrono date_time filesystem python system thread )
FOREACH( _pkg ZLIB gflags glog OpenCV protobuf HDF5 OpenBLAS LevelDB LMDB Snappy )
FIND_PACKAGE( ${_pkg} )
ENDFOREACH()
FIND_PACKAGE( Caffe )

This way, CMake found the paths to the dependencies, but when compiling in Visual Studio I had problems with OpenBLAS and ProtoBuf.

In the case of OpenBLAS, even if CMake found the path to the directory C:/Anaconda2/envs/caffe/Library/lib/cmake/openblas, then when generating the VS solution I see that it tries to link the file OpenBLAS.lib, which doesn't exist (it should be libopenblas.dll.a).

In the case of ProtoBuf in cmake-gui I can see that all the required files were detected with the correct absolute path, but then I'm in VS I see that it tries to link libprotobuf.lib (which is the correct name), but without its path included, so it can't link.

I've been checking the CMake files of each of these dependencies, but I can't figure out what to change to make this work.

[willyd]

Hi @lunzueta I just followed the README.md on a different computer and everything works on my side.

After installing caffe, I do the following (from the SAME console with activated environment):

:: caffe-test contains a dummy CMakeLists.txt
cd caffe-test
mkdir build
cd build
cmake -G"Visual Studio 12 2013 Win64" -DBLAS=Open -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -DCMAKE_MODULE_PATH=%CMAKE_MODULE_PATH% -DCMAKE_PREFIX_PATH=%CMAKE_PREFIX_PATH% -DCMAKE_INSTALL_PREFIX=install ..\
devenv caffe-test.sln

and I get the following libs in the release config:

kernel32.lib
user32.lib
gdi32.lib
winspool.lib
shell32.lib
ole32.lib
oleaut32.lib
uuid.lib
comdlg32.lib
advapi32.lib
C:\Users\guillaume\Documents\Work\caffe\build\lib\caffe.lib
C:\Users\guillaume\Documents\Work\caffe\build\lib\proto.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\boost_python-vc120-mt-1_61.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\boost_thread-vc120-mt-1_61.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\boost_atomic-vc120-mt-1_61.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\boost_chrono-vc120-mt-1_61.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\glog.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\Lib\gflags.lib
shlwapi.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\libprotobuf.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\alt\lib\alt-hdf5_hl.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\alt\lib\alt-hdf5.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\lmdb.lib
ntdll.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\leveldb.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\boost_date_time-vc120-mt-1_61.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\boost_filesystem-vc120-mt-1_61.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\boost_system-vc120-mt-1_61.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\snappy_static.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\alt\lib\alt-zlib.lib
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5\lib\x64\cudart.lib
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5\lib\x64\curand.lib
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5\lib\x64\cublas.lib
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5\lib\x64\cublas_device.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\x64\vc12\lib\opencv_highgui310.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\x64\vc12\lib\opencv_imgcodecs310.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\x64\vc12\lib\opencv_imgproc310.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\x64\vc12\lib\opencv_core310.lib
C:\Users\guillaume\Anaconda2\envs\caffe\Library\lib\libopenblas.dll.a
C:\Users\guillaume\Anaconda2\envs\caffe\libs\python27.lib

Here is my CMakeLists.txt file
CMakeLists.txt

However, I had some trouble with cmake-gui. Make sure the CMAKE_MODULE_PATH and CMAKE_PREFIX_PATH are correctly set in there. I don't know why they don't show up in cmake-gui. I will raise the issue with CMake.

Also, please update to the latest commit as some files were missing from the install.

You can trace CMake behavior with message(STATUS) commands. Look in your anaconda install path for envs\caffe\Library\cmake\modules\FindOpenBLAS.cmake to see if it actually uses this file.