Support to make NVIDIA's Caffe fork (for NVIDIA DIGITS) work in Windows

[lunzueta]

@lukeyeager, I include you here because of your answer in this issue of DIGITS (NVIDIA/DIGITS#47). I'm not sure where on GitHub would be the best place to include this (here, @willyd's Caffe Windows port, NVIDIA's Caffe fork, DIGITS, or somewhere else), but I guess this could be a good place.

Based on @willyd's great Windows port and builder of Caffe (the most recent of the "unofficial Windows ports" cited in Caffe's Wiki: https://github.com/BVLC/caffe/wiki/Installation), I've done a first (unsuccessful) attempt to make NVIDIA's Caffe fork (https://github.com/NVIDIA/caffe) work in Windows too. I guess this is a necessary step to make NVIDIA DIGITS (https://github.com/NVIDIA/DIGITS) work in Windows, as I've seen that it has some extra files not included in this Caffe Windows port.

For that, in the Caffe.cmake file of Caffe-builder I replaced the repository from which Caffe is downloaded with that of NVIDIA's fork. Then, after compiling in VS2013-64-bit all the projects generated by Caffe-builder with NVIDIA's Caffe fork, the only one that failed was that of Caffe (obviously). So, I started checking what the compiler was reporting and based on that I did the following changes:

1. Replaced the file caffe/include/caffe/util/io.hpp from NVIDIA's fork with that of Caffe's Windows port.
2. Added caffe/include/caffe/util/msvc.hpp from Caffe's Windows port.
3. Replaced caffe\src\caffe\layers\bnll_layer.cu with that of Caffe's Windows port.
4. Updated caffe\src\caffe\common.cpp with some Windows-related code lines from that file of Caffe's Windows port.
5. Replaced caffe\src\caffe\layers\contrastive_loss_layer.cpp with that of Caffe's Windows port.

After these changes I could solve some of the initial errors but I still get some that I don't know how to solve:

11>(MY_PATH)\DownloadCache\caffe\src\gtest/gtest.h(670): error C3855: 'std::tuple': template parameter '_Types' is incompatible with the declaration [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>(MY_PATH)\DownloadCache\caffe\src\gtest/gtest.h(670): error C2977: 'std::tuple' : too many template arguments [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>            C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\utility(75) : see declaration of 'std::tuple'
11>(MY_PATH)\DownloadCache\caffe\src\gtest/gtest.h(670): fatal error C1903: unable to recover from previous error(s); stopping compilation [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>    parallel.cpp
11>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(6): fatal error C1083: Cannot open include file: 'sys/ioctl.h': No such file or directory [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>    db.cpp
11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\db.cpp(27): error C3861: 'mkdir': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>    math_functions.cpp
11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\math_functions.cpp(186): warning C4244: 'argument' : conversion from 'const double' to 'const float', possible loss of data [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\math_functions.cpp(346): error C3861: '__builtin_popcount': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\math_functions.cpp(357): error C3861: '__builtin_popcountl': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]

I'm not so experienced in porting so complex projects from Linux to Windows so that's why I got stuck, but my impression is that somebody more experienced could help to accomplish this goal. Can anybody help on this, please?

[willyd]

@lunzueta I am always happy to help when I can. Here is what I can suggest from the error messages above:

• First, be careful with caffe headers if you modify them. There is a flaw in the caffe-builder project because it installs every library in the same folder. Because of this the caffe project includes the installation location in its include directories and will sometimes be confused between the installed caffe and the one you are modifying. When making changes delete the caffe folder in install/include.

11>(MY_PATH)\DownloadCache\caffe\src\gtest/gtest.h(670): error C3855: 'std::tuple': template parameter '_Types' is incompatible with the declaration [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>(MY_PATH)\DownloadCache\caffe\src\gtest/gtest.h(670): error C2977: 'std::tuple' : too many template arguments [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11> C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\utility(75) : see declaration of 'std::tuple'

I added a preprocessor definition in the root CMakeLists.txt for this. Make the same change.

11>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(6): fatal error C1083: Cannot open include file: 'sys/ioctl.h': No such file or directory [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11> db.cpp

This one I don't remember ever seeing this.

11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\db.cpp(27): error C3861: 'mkdir': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11> math_functions.cpp

You need a define:

#define mkdir _mkdir

11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\math_functions.cpp(346): error C3861: '__builtin_popcount': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]
11>(MY_PATH)\DownloadCache\caffe\src\caffe\util\math_functions.cpp(357): error C3861: '__builtin_popcountl': identifier not found [(MY_PATH)\Caffe-prefix\src\Caffe-build\src\caffe\caffe.vcxproj]

Same here. You need a define for these two. See msvc.hpp.

HTH

[lunzueta]

Thanks for the hints @willyd. After applying your suggestions, the only compilation error I get now (apart from some linking errors that come in the end which I guess I still can't solve until I fix this), is that related to 'sys/ioctl.h'. I guess you didn't see this before because it seems to be something specific to NVIDIA's fork (in file caffe\src\caffe\parallel.cpp). Any idea about what to do with this? I'll search about what to do for porting this to Windows with the minimal possible changes, but this would be my first time with this file, so any suggestion from somebody more experienced is welcome.

[willyd]

@lunzueta I am not so experienced in porting code from Linux to Windows myself either. So your guess is as good as mine. Google is your friend.

[lunzueta]

I got a bit stuck with this now. In the parallel.cpp file there are inclusions of two header files which are specific to Linux: sys/ioctl.h and sys/mman.h. If you just skip them and compile the parallel.cpp file (to see which functions rely on those inclusions) you don't get any error but just the following warnings:

1>  parallel.cpp
1>(MY_PATH)\install\include\H5public.h(158): warning C4091: 'typedef ' : ignored on left of '__int64' when no variable is declared
1>(MY_PATH)\DownloadCache\caffe\include\caffe/loss_layers.hpp(16): warning C4305: 'initializing' : truncation from 'double' to 'const float'
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(156): warning C4244: '=' : conversion from 'double' to 'int', possible loss of data
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(181): warning C4244: '=' : conversion from 'double' to 'int', possible loss of data
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(98): warning C4267: 'argument' : conversion from 'size_t' to 'const int', possible loss of data
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(84) : while compiling class template member function 'caffe::GPUParams<float>::GPUParams(boost::shared_ptr<caffe::Solver<Dtype>>,int)'
1>          with
1>          [
1>              Dtype=float
1>          ]
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(512) : see reference to class template instantiation 'caffe::GPUParams<float>' being compiled
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(318): warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(296) : while compiling class template member function 'void caffe::P2PSync<float>::on_start(caffe::Timer *,std::ostringstream *)'
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(513) : see reference to class template instantiation 'caffe::P2PSync<float>' being compiled
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(372): warning C4267: 'argument' : conversion from 'size_t' to 'const int', possible loss of data
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(342) : while compiling class template member function 'void caffe::P2PSync<float>::on_gradients_ready(caffe::Timer *,std::ostringstream *)'
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(398): warning C4267: 'argument' : conversion from 'size_t' to 'const int', possible loss of data
1>(MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(290): warning C4244: 'argument' : conversion from 'google::protobuf::int64' to 'const unsigned int', possible loss of data
1>          (MY_PATH)\DownloadCache\caffe\src\caffe\parallel.cpp(280) : while compiling class template member function 'void caffe::P2PSync<float>::InternalThreadEntry(void)'
1>C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xutility(2132): warning C4996: 'std::_Copy_impl': Function call with parameters that may be unsafe - this call relies on the caller to check that the passed values are correct. To disable this warning, use -D_SCL_SECURE_NO_WARNINGS. See documentation on how to use Visual C++ 'Checked Iterators'
1>          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\xutility(2113) : see declaration of 'std::_Copy_impl'
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1293) : see reference to function template instantiation '_OutIt std::copy<const Element*,Element*>(_InIt,_InIt,_OutIt)' being compiled
1>          with
1>          [
1>              _OutIt=google::protobuf::int64 *
1>  ,            Element=google::protobuf::int64
1>  ,            _InIt=const google::protobuf::int64 *
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1292) : while compiling class template member function 'void google::protobuf::internal::ElementCopier<Element,false>::operator ()(Element *,const Element *,int)'
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1285) : see reference to function template instantiation 'void google::protobuf::internal::ElementCopier<Element,false>::operator ()(Element *,const Element *,int)' being compiled
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1285) : see reference to class template instantiation 'google::protobuf::internal::ElementCopier<Element,false>' being compiled
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1284) : while compiling class template member function 'void google::protobuf::RepeatedField<google::protobuf::int64>::CopyArray(Element *,const Element *,int)'
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1279) : see reference to function template instantiation 'void google::protobuf::RepeatedField<google::protobuf::int64>::CopyArray(Element *,const Element *,int)' being compiled
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1278) : while compiling class template member function 'void google::protobuf::RepeatedField<google::protobuf::int64>::MoveArray(Element *,Element *,int)'
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1254) : see reference to function template instantiation 'void google::protobuf::RepeatedField<google::protobuf::int64>::MoveArray(Element *,Element *,int)' being compiled
1>          with
1>          [
1>              Element=google::protobuf::int64
1>          ]
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1223) : while compiling class template member function 'void google::protobuf::RepeatedField<google::protobuf::int64>::Reserve(int)'
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1065) : see reference to function template instantiation 'void google::protobuf::RepeatedField<google::protobuf::int64>::Reserve(int)' being compiled
1>          (MY_PATH)\install\src\google/protobuf/repeated_field.h(1010) : while compiling class template member function 'int google::protobuf::RepeatedField<google::protobuf::int64>::size(void) const'
1>          (MY_PATH)\Caffe-prefix\src\Caffe-build\include\caffe/proto/caffe.pb.h(9037) : see reference to function template instantiation 'int google::protobuf::RepeatedField<google::protobuf::int64>::size(void) const' being compiled
1>          (MY_PATH)\Caffe-prefix\src\Caffe-build\include\caffe/proto/caffe.pb.h(686) : see reference to class template instantiation 'google::protobuf::RepeatedField<google::protobuf::int64>' being compiled

From this info, I don't see clearly what to do next, but I guess something has to be done instead of just skipping those inclusions. Don't know...

Anyway, from this point on, if I compile the whole caffe.vcxproj project, I don't get any error either. The caffe-nv.lib file (caffe.lib in your Windows port, @willyd) is generated in Caffe-prefix\src\Caffe-build\lib\Release folder (caffe-nv-d.lib in Caffe-prefix\src\Caffe-build\lib\Debug folder), but those files aren't automatically copied to the install/lib folder, as it happens with caffe.lib and caffe-d.lib in your port. What should be changed to make the builder copy automatically those files to install/lib? Something related to CMake files?

[willyd]

You need to build the INSTALL project. Equivalent of make install on Linux.

[lunzueta]

Thanks @willyd, that worked for the copy. Anyway, I see that more changes have to be done to make this work. I've done another fork of NVIDIA's fork to have a version control of the changes, and to be able to share this with others: https://github.com/lunzueta/caffe

Anyone interested in collaborating on this, let me know, so that I can give access.

[lunzueta]

@willyd, I have one question about how to adapt your super builder to make it work with my fork. In the file Caffe.cmake I did the following change:

Instead of this:

buildem_download_package(GIT_REPOSITORY "https://github.com/willyd/caffe.git"
                         GIT_TAG visualstudio
                         SOURCE_DIR Caffe_SOURCE_DIR)

I put this:

buildem_download_package(GIT_REPOSITORY "https://github.com/lunzueta/caffe.git"
                         SOURCE_DIR Caffe_SOURCE_DIR)

I'm not sure what GIT_TAG is for, because I still didn't find a good explanation about it on the Internet. In my repository there's not visualstudio branch, so I just removed that. Would that be ok?

[willyd]

@lunzueta GIT_TAG stands for anything you would do:

git checkout ${GIT_TAG}

so a branch, a commit sha-1 or a tag. You can also remove GIT_TAG altogether so that the default branch (usually master) is checked out.

[lunzueta]

Ok, thanks. Understood. In my case, for now, it's simpler just to remove it.

[lunzueta]

At this moment, I'm able to compile NVCaffe from my fork (https://github.com/lunzueta/caffe) in Release mode, except the projects "runtest" and "test.testbin", but I guess I can start testing DIGITS with the compiled executables and libraries. I'll report about these tests when possible.

Anyway, I'd like to be able to compile all projects in Release and also in Debug to finish properly this port. The errors I get are the following:

runtest and test.testbin projects (Release):

2>(MY_PATH)\install\include\caffe/test/test_gradient_check_util.hpp(165): error C2784: '_Ty std::max(std::initializer_list<_Elem>,_Pr)' : could not deduce template argument for 'std::initializer_list<_Elem>' from 'const float'
2>          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\algorithm(4108) : see declaration of 'std::max'
2>(MY_PATH)\install\include\caffe/test/test_gradient_check_util.hpp(165): error C2780: 'const _Ty &std::max(const _Ty &,const _Ty &,_Pr)' : expects 3 arguments - 2 provided

pycaffe (Debug):

2>LINK : fatal error LNK1104: cannot open file 'python27_d.lib'

The linking error of pycaffe in Debug is strange because python27_d.lib doesn't appear in the project linking properties as a dependence, but python27.lib. In @willyd's equivalent project this project doesn't have this issue in Debug.

Any suggestions?

[willyd]

2>LINK : fatal error LNK1104: cannot open file 'python27_d.lib'

You need to #include boost/python/detail/wrap_python.hpp instead of #include <python.h>. See http://www.boost.org/doc/libs/1_58_0/libs/python/doc/building.html#python-debugging-builds for details.

2>(MY_PATH)\install\include\caffe/test/test_gradient_check_util.hpp(165): error C2784: '_Ty std::max(std::initializer_list<_Elem>,_Pr)' : could not deduce template argument for 'std::initializer_list<_Elem>' from 'const float' 2> C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\algorithm(4108) : see declaration of 'std::max' 2>(MY_PATH)\install\include\caffe/test/test_gradient_check_util.hpp(165): error C2780: 'const _Ty &std::max(const _Ty &,const _Ty &,_Pr)' : expects 3 arguments - 2 provided

std::max is a template function. It seems MSVC cannot deduce the type it should use. You have to specify it manually like so std::max<Type>(arg1, arg2).

[lunzueta]

Thanks @willyd, the change about python worked. Now pycaffe compiles correctly in Debug too.

Regarding the template issue, I'm not sure how to apply what you tell me. The only differences I see in test_gradient_check_util.hpp in Nvidia's fork with respect to yours are the following two:

1. In your file:

// First, figure out what blobs we need to check against.
  vector<Blob<Dtype>*> blobs_to_check;
  vector<bool> propagate_down(bottom.size(), check_bottom < 0);
  for (int i = 0; i < layer->blobs().size(); ++i) {
    blobs_to_check.push_back(layer->blobs()[i].get());
  }

In Nvidia's fork:

  // First, figure out what blobs we need to check against, and zero init
  // parameter blobs.
  vector<Blob<Dtype>*> blobs_to_check;
  vector<bool> propagate_down(bottom.size(), check_bottom < 0);
  for (int i = 0; i < layer->blobs().size(); ++i) {
    Blob<Dtype>* blob = layer->blobs()[i].get();
    caffe_set(blob->count(), static_cast<Dtype>(0), blob->mutable_cpu_diff());
    blobs_to_check.push_back(blob);
  }

2. In your file:

  Dtype scale = std::max(
    std::max(fabs(computed_gradient), fabs(estimated_gradient)), 1.);

In Nvidia's fork:

  Dtype scale = std::max<Dtype>(
    std::max<Dtype>(fabs(computed_gradient),
    fabs(estimated_gradient)), 1.);

The rest of the file is exactly the same in both sides. What do you think would be the best to solve this?

[willyd]

@lunzueta

In Nvidia's fork:

Dtype scale = std::max(
std::max(fabs(computed_gradient),
fabs(estimated_gradient)), 1.);

These are my changes not Nvidia's. Weirdly enough github's markdown removes the <Dtype> in the quotation :(.

Something is confusing the compiler type deduction. There are multiple overloads of std::max and this is why you have a cryptic error with an std::initializer_list. My guess is that since Dtype can be of type float or double and that 1. is double by default the compiler cannot deduce if it should use std::max<float> or std::max<double>. To get around this you have to use my version which forces the usage of std::max<Dtype> or try this code instead:

 Dtype scale = std::max(
    std::max(fabs(computed_gradient),
    fabs(estimated_gradient)), Dtype(1.));

[lunzueta]

Thanks @willyd. This last change worked. Now I can compile all the NVCaffe projects in Release and Debug in VS2013 and x64. I guess this is a good moment to start testing Digits with this and see if everything is ok :)

[lunzueta]

@willyd, while testing the generated NVCaffe executable for training the MNIST data, I've seen that I get this failure, which I don't get with your fork (your fork does this correctly):

F0913 13:37:23.739025  7344 layer_factory.hpp:79] Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: Data (known types: )
*** Check failure stack trace: ***

As you can see it doesn't find the known layer types properly. It's not straightforward to figure out where this behavior can come from. While you were doing your Windows port, did you find this kind of behavior at some point (and fixed it)?