diff --git a/README.md b/README.md index 7d0ec89..48b2d5b 100644 --- a/README.md +++ b/README.md @@ -1 +1,24 @@ -# spark-mpi-tf \ No newline at end of file +# SPARK-MPI-TF + +This project demostrates the Spark-MPI approach within the context of Spark-based TensorFlow distributed deep learning +applications. The direction is addressed by several other projects, such as +[BigDL](https://github.com/intel-analytics/BigDL) and +[TensorFlowOnSpark](https://github.com/yahoo/TensorFlowOnSpark). In comparison with these alternative +solutions, Spark-MPI aims to derive an application-neutral mechanism based on the MPI Process Management Interface (PMI) +for the effortless integration of Big Data and HPC ecosystems. + +## Prerequisites + +1. [Spark-MPI](https://github.com/SciDriver/spark-mpi): PMI-based approach for integrating together the Spark platform and MPI applications + +2. [Horovod](https://github.com/uber/horovod): MPI-based training framework for TensorFlow + +## Examples + +The MNIST Spark-Horovod [IPython notebook](https://github.com/SciDriver/spark-mpi-tf/blob/master/examples/mnist/spark_horovod.ipynb) for handwritten digit classification (see, for reference, [TensorFlow Tutorial](https://www.tensorflow.org/tutorials/layers)). + + + + + + diff --git a/examples/mnist/README.txt b/examples/mnist/README.txt index 5c999ef..228b51e 100644 --- a/examples/mnist/README.txt +++ b/examples/mnist/README.txt @@ -1,6 +1,9 @@ +This Spark-MPI example demonstrates the integration of Horovod's MPI-based deep learning engine +with the Spark platform within the context of the MNIST application. -Running Spark-MPI with Jupyter -------------------------------- + +Running with Jupyter on a single node +------------------------------------- export HYDRA_PROXY_PORT=55555 @@ -9,4 +12,3 @@ export PYSPARK_DRIVER_PYTHON_OPTS='notebook --no-browser --port=7777' pyspark --master local[*] -pkill -9 "hydra_pmi_proxy"