Multinode over internet, Async and Parallel SGD, WebAssembly

[bhack]

I see that your are opening here also to a distributed perspective. What do you think of some emerging solution like http://arxiv.org/abs/1503.05743? Yes this rely on WebCL that it is not supported natively in any browser and actually propose only convolutional layers in a distributed fashion. But some discussion on WebAssembly, in which also Google is involved, are starting to think on GPU support of this forming standard. See WebAssembly/design#273. Do you think that caffe2 design could be further proof to scale at the level where every node could run on a browser tab? Or don't you believe that training and networks design could evolve at large scale relying on users nodes with "standard" internet bandwidth interconnection?

[Yangqing]

Thanks - yeah, I think as long as we stick to the same protobuf definition (sort of like API) this would be possible. Of course one needs to write the WebCL implementation (Andrej actually wrote one before but with custom format) - other than that it should just be a formatting issue.

[Yangqing]

Feel free to tinker around with it, but this is honestly not a top
priority...

Sent from Gmail mobile
On Aug 30, 2015 9:29 AM, "bhack" notifications@github.com wrote:

How do you think that a JavaScript back end could be introduced in the
design? I.e. starting from the JavaScript matrix component
https://github.com/mil-tokyo/sushi

—
Reply to this email directly or view it on GitHub
https://github.com/Yangqing/caffe2/issues/11#issuecomment-136157348.

[bhack]

@Yangqing Yes I think that browser technology it is not enough mature actually so cannot be a priority for caffe2. But I think that if WebAssembly go through covering GPU computing could be interesting to have a "compatible" design . Can I ask you what are your design targets on distribute the training and parameters update? On caffe multigpu is oriented mainly on fast p2p communication between GPU on PCI bus. Are you targeting also multi node? On which kind of network infrastructures?

[bhack]

Some interesting Caffe experiments with Multicore/MultiGPU and Cluster nodes at http://arxiv.org/abs/1506.08272

[bhack]

See also BVLC/caffe#1148 (comment)