Tutorial and codes for Parallel Computing (MPI)
Download Package Manually: https://bitbucket.org/mpi4py/mpi4py/downloads/?tab=downloads
Official MPI Website: https://mpi4py.readthedocs.io/en/stable/install.html
Installation on UBUNTU: http://selalib.gforge.inria.fr/install/ubuntu.html
Official Pypi website: https://pypi.org/project/mpi4py/
Sentdex nstalling mpi4py for use with Python and MPI: https://pythonprogramming.net/installing-testing-mpi4py-mpi-python-tutorial/
MPI Website Official Tutorial: https://mpi4py.readthedocs.io/en/stable/tutorial.html
PDF from Yale University: https://research.computing.yale.edu/sites/default/files/files/mpi4py.pdf
Advanced MPI Slides Pittsburg University (C++): https://www.psc.edu/images/xsedetraining/MPI/Advanced_MPI.pdf
A Very Nice Slide for MPI4Py (Recommended++): https://portal.tacc.utexas.edu/documents/13601/1102030/4_mpi4py.pdf/f43b984e-4043-44b3-8225-c3ce03ecb93b
A very nice website for MPI4PY (Recommended++): https://nyu-cds.github.io/python-mpi/05-collectives/
Github Repo with demo codes: https://github.com/mpi4py/mpi4py/tree/master/demo
A Very Nice Website for the Tree and Classes of MPI4PY (Highly Recommended): http://eups.linea.gov.br/external/mpi4py/mpi4py-1.3/docs/apiref/mpi4py.MPI-module.html
Nice MPI4py Slide (HPC): https://www.hi-mainz.de/fileadmin/user_upload/IT/lectures/WiSe2019/HPC/files/Lecture_HPC_8_MPI_Python.pdf
A very Nice website on MPI and its structure (C++)-Recommended: https://computing.llnl.gov/tutorials/mpi/
A very nice Gihub Repo for MPI+HPC+... (Recommended): https://gitlab.rlp.net/pbotte/learnhpc/-/tree/master/
Website MPI (C++): https://pages.tacc.utexas.edu/~eijkhout/pcse/html/mpi-ptp.html#Bufferissuesinnon-blockingcommunication
Asynchronous Communication (C++): http://www.mathcs.emory.edu/~cheung/Courses/561/Syllabus/92-MPI/async.html
Communications in MPI (C++): https://gcl.cis.udel.edu/courses/cisc849010_sp09/secure/data_gp/02L_MPI_01_solution.pdf
Asynchronous communication (C++): http://supercomputingblog.com/mpi/mpi-tutorial-5-asynchronous-communication/
Sentdex MPI: https://pythonprogramming.net/basic-mpi4py-script-getting-node-rank/
MPI4Py nice tutorial website: https://www.kth.se/blogs/pdc/2019/08/parallel-programming-in-python-mpi4py-part-1/
Non Blocking Communication (C++): https://www.codingame.com/playgrounds/349/introduction-to-mpi/non-blocking-communications---exercise
Non-Blocking Communication: https://nyu-cds.github.io/python-mpi/03-nonblocking/
Collective Operations: https://nyu-cds.github.io/python-mpi/05-collectives/
MPI4py crash course Tutorials Slides: http://www.ceci-hpc.be/assets/training/mpi4py.pdf
MPI for Python (PDF version of official website): https://buildmedia.readthedocs.org/media/pdf/mpi4py/stable/mpi4py.pdf
MPI More of the Story very nice slides (Recommended): http://geco.mines.edu/prototype/Show_me_some_local_HPC_tutorials/mpi4py/mpi02_py.pdf
MPI Very Nice Slides (Highly-Recommended): https://mosaic.mpi-cbg.de/docs/DCAMM/02_Tuesday_1.pdf
MPI for Python nice slides: https://www.bu.edu/pasi/files/2011/01/Lisandro-Dalcin-mpi4py.pdf
Parallel programming in Python: mpi4py (part 2): https://www.kth.se/blogs/pdc/2019/11/parallel-programming-in-python-mpi4py-part-2/
A nice website: https://cvw.cac.cornell.edu/python/mpi4py
CSE 260 (HPC course)- Higly Recommended: http://www.cse.ucsd.edu/classes/wi08/cse260
CSE 260 first Lecture PDF: http://cseweb.ucsd.edu/classes/wi08/cse260/Lectures/Lec01/Lec01.pdf
CSE 260 Isend/Irecv Lecture: http://cseweb.ucsd.edu/classes/fa06/cse260/Lectures/Lec03/Lec03.pdf
CSE 260 MPI Information: https://cseweb.ucsd.edu/classes/sp99/cse160/programming/mpi_info/mpi_info.html
CSE 260 On line resources - publications, documentation, software, courses, etc.: https://cseweb.ucsd.edu/~baden/Doc/#par-sci
CSE 260 MPI: https://cseweb.ucsd.edu/~baden/Doc/mpi.html
CSE 164B: http://cseweb.ucsd.edu/classes/fa98/cse164b/HomePage/descrip.html
MPI (C++) Datatypes: https://www.mpi-forum.org/docs/mpi-2.1/mpi21-report-bw/node330.htm
Nice MPI slides: https://janth.home.xs4all.nl/MPIcourse/PDF/03_MPI_BasicsHandOut.pdf
Parallel Programming in MPI and OpenMP Book: http://enformtk.u-aizu.ac.jp/courses/hpc/EijkhoutParComp.pdf
MPI Datatypes: https://www.mpi-forum.org/docs/mpi-2.1/mpi21-report-bw/node330.htm
MPI Fork Issues: http://gfs.sourceforge.net/wiki/index.php/System(),_popen(),_fork()_and_OpenMPI
Slides from Cornell University: http://www.cac.cornell.edu/education/training/StampedeJune2013/IntroMPI-130611.pdf
Introduction to parallel programming and MPI: https://software.rc.fas.harvard.edu/training/Intro_to_MPI.pdf
Introduction to MPI-- Texas Advanced Computing Center: https://portal.tacc.utexas.edu/c/document_library/get_file?uuid=2d346261-1555-45ae-b404-72ff289e2986&groupId=13601
Nice slides on Python in HPC: https://events.prace-ri.eu/event/806/attachments/833/1154/PRACE-PTC_Python_in_HPC__CSC_23.-25.1.19.pdf
PACE: Introduction to parallel computing (Higly Recommended): https://pace.gatech.edu/sites/default/files/pace_intro_to_parallelcomp.pdf
MPI: Basic send and recv: https://janth.home.xs4all.nl/MPIcourse/PDF/03_MPI_Basics.pdf
MPI Communication (ping-pong, ...) From SDSU (Recommended): https://edoras.sdsu.edu/~mthomas/sp17.605/lectures/MPI-Comms-Perf.pdf
Hybrid MPI and OpneMP parallel programming: https://openmp.org/wp-content/uploads/HybridPP_Slides.pdf
Introduction to MPI: https://courses.cs.washington.edu/courses/csep524/13wi/Thakur-MPI.pdf
ADVANCEDMPI 2.2 AND3.0 TUTORIAL--Highly Recommended: https://htor.inf.ethz.ch/teaching/mpi_tutorials/cscs12/hoefler_tutorial_advanced-mpi-2.2-and-mpi-3.0_cscs.pdf
YCRC Bootcamp: Python MPI for Parallel Programming: https://www.youtube.com/watch?v=ZUFgEovC_WQ
Parallel Programming in Python Using MPI: https://www.youtube.com/playlist?list=PLagFkXs2BczZO4B_I6YhuS4xOxlSPgZuT
Introduction to parallel Programming -- Message Passing Interface (MPI): https://www.youtube.com/watch?v=aqySX4lpzn0
Sentdex Supercomputing and Parallel Programming in Python and MPI: https://www.youtube.com/watch?v=GvNiP1yfbsc&list=PLQVvvaa0QuDf9IW-fe6No8SCw-aVnCfRi&index=18
Sentdex Asyncio - Asynchronous programming with coroutines - Intermediate Python Programming: https://www.youtube.com/watch?v=BI0asZuqFXM
Parallel Implementation of Deep Learning Using MPI: https://cse.buffalo.edu/faculty/miller/Courses/CSE633/Ma-Spring-2014-CSE633.pdf
Asynchronous Parallel Methods: http://www.archer.ac.uk/training/course-material/2018/04/adv-mpi-exeter/Slides/L08-Asynchronous.pdf
Advanced MPI: https://htor.inf.ethz.ch/teaching/mpi_tutorials/isc16/hoefler-balaji-isc16-advanced-mpi.pdf
Advanced Parallel Programming withMPI-1, MPI-2, and MPI-3: https://htor.inf.ethz.ch/teaching/mpi_tutorials/ppopp13/2013-02-24-ppopp-mpi-advanced.pdf
Some Examples (C++): https://github.com/hpc/MPI-Examples
Torch MPI: https://github.com/facebookarchive/TorchMPI
Very Nice Sample Codes (C++) -- Highly Recommended: https://github.com/wesleykendall/mpitutorial
Nice mpi4py example codes: https://github.com/jbornschein/mpi4py-examples
A nice repo (MPI Tutorials): https://github.com/wesleykendall/mpitutorial/tree/gh-pages
Webpage of the above Repo: https://mpitutorial.com/tutorials/point-to-point-communication-application-random-walk/
Introduction to parallel computing (async_ping_pong)--Highly Recommended: https://pace.gatech.edu/sites/default/files/pace_intro_to_parallelcomp.pdf
Ping Pong communication Repo--Recommended: https://github.com/hpcugent/mympingpong/tree/master/lib/vsc/mympingpong
Avoiding DeadLocks (Highly Recommended): https://courses.cs.ut.ee/MTAT.08.020/2019_fall/uploads/Main/MPI_p2p-slides.pdf
https://www.futurelearn.com/courses/python-in-hpc/0/steps/65147
https://www.csc.fi/documents/200270/224572/advanced_MPI.pdf/3a49a631-d403-4bbf-85ba-a8daf17f17d2
Communication Completion: https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-1.1/node47.htm
Probe and Cancel: https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-1.1/node50.htm#Node50
Multiple Completions: https://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-1.1/node49.htm#Node49
https://www.mpich.org//static/docs/latest/
http://eups.linea.gov.br/external/mpi4py/mpi4py-1.3/docs/apiref/mpi4py.MPI.Request-class.html
https://www.open-mpi.org/doc/v4.0/
A very nice Book (Recommended): http://www.sas.rochester.edu/psc/thestarlab/help/MPI_Course.pdf
An Introduction to MPI: https://www.mjr19.org.uk/courses/MPI/MPI.pdf
An Introduction to parallel computing: http://www.e-tahtam.com/~turgaybilgin/2013-2014-guz/ParalelProgramlama/ParallelProg.pdf
Message Passing Fundamentals (Recommended): http://www.sas.rochester.edu/psc/thestarlab/help/MPI_Course.pdf
Parallel Programming in MPI and OpenMP: http://faculty.salisbury.edu/~jtanderson/teaching/cosc420/fa19/files/mpi.pdf
MPI Course--Higly Recommended: http://www.sas.rochester.edu/psc/thestarlab/help/
MPI: A Message-Passing Interface Standard2019 Draft Specification--Higly Recommended: https://www.mpi-forum.org/docs/drafts/mpi-2019-draft-report.pdf
https://bitbucket.org/mpi4py/mpi4py/issues/148/_pickleunpicklingerror-could-not-find-mark
https://bitbucket.org/mpi4py/mpi4py/issues?status=invalid
[mpi4py] Problem using isend/recv to send/receive large data structure: https://groups.google.com/forum/#!searchin/mpi4py/isend$20irecv$20|sort:date/mpi4py/UlR2y7yQ9Fc/wqX0Fa-oCwAJ
irecv().test() errors. corrupted pickles? (Important): https://groups.google.com/forum/#!searchin/mpi4py/isend$20irecv$20|sort:date/mpi4py/fEsr9cOcohE/asGhI7SdAQAJ
message truncation error when passing numpy array to recv() (Important): https://groups.google.com/forum/#!searchin/mpi4py/isend$20irecv$20|sort:date/mpi4py/CrYXfzm7E0Y/GAVYh0YuEBwJ
Experiences and Hints: https://groups.google.com/forum/#!searchin/mpi4py/isend$20irecv$20|sort:date/mpi4py/pp4wiwLWIMU/jkY0OR-iDVAJ
[mpi4py] how to best transfer image data, i.e. header + 2d array (Important): https://groups.google.com/forum/#!searchin/mpi4py/asynchronous|sort:date/mpi4py/uIfjIhkUxm4/G2ZjRedE6_AJ
MPI_ERR_TRUNCATE: message truncated when using Comm.irecv() with messages larger than 32kb: https://bitbucket.org/mpi4py/mpi4py/issues/65/mpi_err_truncate-message-truncated-when
MPI Forum Github: https://github.com/mpi-forum/mpi-forum.github.io
Jeff Hammond's BigMPI:
Multi GPU, multi process with Tensorflow (Snake Game): https://towardsdatascience.com/multi-gpu-multi-process-with-tensorflow-ba4cc2fe3ab7
Tensorflow GPU Guide: https://www.tensorflow.org/guide/gpu