This project demostrates the Spark-MPI approach within the context of Spark-based TensorFlow distributed deep learning applications. The direction is addressed by several other projects, such as BigDL and TensorFlowOnSpark. In comparison with these alternative solutions, Spark-MPI aims to derive an application-neutral mechanism based on the MPI Process Management Interface (PMI) for the effortless integration of Big Data and HPC ecosystems.
-
Spark-MPI: PMI-based approach for integrating together the Spark platform and MPI applications
-
Horovod: MPI-based training framework for TensorFlow
The MNIST Spark-Horovod IPython notebook for handwritten digit classification (see, for reference, TensorFlow Tutorial).