Split CUDA code (*.cu) from CPU code (*.cpp).

[sergeyk]

This will enable CPU-only Caffe compilation (#3), and go a long way to 10.9 CUDA compilation problems (#122)

Checklist [shelhamer]:

• split layers into cpp and cu Splitting source files between CUDA and CPU code. #172
• split enough for boost-eigen to build on osx (hide boost rng from nvcc) MKL/non-MKL Reconciliation #165
• split and tag tests as CPU / GPU
• split math_functions into cpp and cu
• split common into normal and cpu-only singletons?

Follow-up: abstract CPU / GPU device computation #610

[tdomhan]

What's the plan to go about this? Split each layer into two subclasses, e.g. ConvLayerCPU and ConvLayerGPU or then have dummy Foward_GPU functions in the *.cpp files?

[sergeyk]

@erictzeng would be good if you submitted a work-in-progress pull request for this, or shared your plan here so that others could provide feedback before too much work is done

[shelhamer]

Sorry, we talked off-list about this. He's waiting for the merge of #142 and #163 but already made the split and it builds and tests pass.

@erictzeng it'd be good to push and PR your current work to dev anyway, and we can look at it once the mentioned PRs are folded in.

[shelhamer]

Closing since #172 is in.

[kloudkl]

The discussion in #172 indicates that a CPU only version is only possible when the CPU and CUDA codes live in different classes or better different namespaces like what was done in OpenCV. Do you agree?

[tdomhan]

@kloudkl what was it exactly that the opencv folks did? Something like having a namespace opencv_cpu and opencv_gpu?

[kloudkl]

https://github.com/Itseez/opencv

[kloudkl]

The modules dir contains cuda and cudev namespaces to separate the GPU related codes. The platforms dir includes cmake files and build scripts specific to the most common platforms.

[shelhamer]

Splitting common into cpp and cuda code with a common interface that satisfies the osx issue in #165 turns out to be intricate. An immediate frustration is that the Caffe singleton itself relies on curand and cublas, and although random number generation and blas can be abstracted into strategies for cpu/gpu operation this doesn't handle 1. switching modes or 2. mixed cpu+gpu implementations.

Instead of totally splitting, the simplest approach could be to implement two Caffe singletons

1. the normal Caffe singleton, already coded in common.cpp
2. a cpu-only Caffe singleton, to be coded in common_cpu.cpp

that both obey the same interface. However, the cpu-only singleton will have no-ops for gpu methods and instead complain through warnings when it is passed gpu methods are called or gpu args are passed. In this way the rest of Caffe such as the solver, tools, and the like can stay the same. Instead of ifdefs all around the changes will be isolated to the Caffe singletons.

[shelhamer]

Done in #561. #610 will carry it further to simplify the code and pave the way for different hardware backends.