################################################################################

This distribution contains the source code for mpimemu, a simple tool that helps
approximate MPI library memory usage as a function of scale.

################################################################################

########################################
## Quick Start
########################################
Please see README.QUICKSTART.

########################################
## Latest Distributions
########################################
http://hpc.github.com/mpimemu

########################################
## Installation
########################################

First make certain that mpicc or another MPI wrapper compiler with similar
capabilities is in your $PATH.

o Standard Configure -- mpicc and cc will be checked by default.

./configure

o Configure with Different Wrapper Compiler

./configure CC=[NEWCC]

o Build

make && make install

*** Example Configure and Build

./configure --prefix=$HOME/local/mpimemu && make && make install

o Installation Notes

mpimemu does not necessarily need to be installed (via "make install") -- it can
be used from within its source directory.

########################################
## Environment Setup
########################################

mpimemu-run is a helper utility used to run a succession of mpimemu instances
of various sizes. The following environment variables change the way
mpimemu-run behaves.

o Required Environment Variables

- MPIMEMU_MAX_PES: Specifies the maximum number of MPI processes that will be
                   launched.

- MPIMEMU_RUN_CMD: Specifies a "run template" that will be used to launch
                   mpimemu jobs.

                   *** Example Run Template with mpirun
                       mpirun -n nnn aaa
                       mpirun -n nnn -npernode NNN aaa

                   *** Example Run Template with aprun
                       aprun -n nnn aaa
                       aprun -n nnn -N NNN aaa

                   * Required Template Variables
                     - nnn: Replaced with the total number processes to be
                            launched. This value changes after each run and is
                            determined by MPIMEMU_NUMPE_FUN(X). More on
                            MPIMEMU_NUMPE_FUN below.
                     - aaa: Replaced with the mpimemu run string.

                   * Optional Template Variables
                     - NNN: Replaced with MPIMEMU_PPN. If MPIMEMU_PPN is not
                            set, 1 will be used.

- MPIMEMU_NUMPE_FUN: Specifies the function that determines how nnn grows with
                     respect to X. "X" must be provided within the string
                     defining the function.

                     *** Example (bash-like)
                     export MPIMEMU_START_INDEX=1
                     export MPIMEMU_NUMPE_FUN="X + 1"
                     export MPIMEMU_MAX_PES=4
                     Will run jobs of size: 1, 2, 3, 4.

                     * Useful, Accepted Arithmetic Operators
                       +   Addition
                       *   Multiplication
                       **  Exponentiation

o Optional Environment Variables

- MPIMEMU_START_INDEX: Specifies the starting integer value of an increasing
                       value, X.  When set, X is incremented by 1, starting at
                       the specified value, while MPIMEMU_NUMPE_FUN(X) <=
                       MPIMEMU_MAX_PES.

- MPIMEMU_PPN: Specifies the number of MPI processes per node.

- MPIMEMU_DATA_DIR_PREFIX: Specifies the base directory where mpimemu data is
                           stored.

- MPIMEMU_BIN_PATH: Specifies the directory where mpimemu is located.

- MPIMEMU_SAMPS_PER_S: Integer value specifying the number of samples that will
                       be collected every second.
                       Default: 10 samples per second.

- MPIMEMU_SAMP_DURATION: Integer value specifying the sampling duration in
                         seconds.
                         Default: 10 seconds.

- MPIMEMU_DISABLE_WORKLOAD: If set, disables synthetic MPI communication
                            workload.

!!! Please note that mpimemu requires that all nodes have the same number of
tasks. This implies that for all values of MPIMEMU_NUMPE_FUN(X), the task per
node requirement must be maintained.  Otherwise, some run failures may occur.

########################################
## Running
########################################

Once your environment is setup, add your mpimemu installation prefix to your
$PATH or run from within the source distribution.

*** Example (if in $PATH)
mpimemu-run

*** Example (from within source distribution)
./src/mpimemu-run

<lots of output>

When complete, the data directory will be echoed.
For example:
data written to: /users/samuel/mpimemu-samuel-01032013

########################################
## Generating CSV Files
########################################

mpimemu-mkstats consolidates data generated by mpimemu-run. To generate CSV
files, simply run mpimemu-mkstats -i [/path/to/data]. For more options and
information, run mpimemu-mkstats -h.

########################################
## Plotting the Data (CSVs)
########################################
mpimemu-plot can be used to generate node and process memory usage from output
generated by mpimemu-mkstats. gnuplot and ps2pdf are required to be in your
PATH.

########################################
## Interpreting the Data
########################################
Take idle system memory usage into consideration (e.g system image size).
mpimemu can provide a general sense of memory usage scaling, but it is important
to note that not all the memory being consumed on a compute resource is due to
MPI library usage. Generally it is a good idea to start your scaling runs at 1
MPI task to get a general sense about "close-to-base" memory usage (i.e base
system usage).

mpimemu-plot presents MPI memory usage as (MemUsed - Pre Init MemUsed). Please
feel free to contact me if you have a better metric.

## FAQ

Why is configure failing with:
[X] cannot compile MPI applications. cannot continue.
Try the following if using mpicc.
./configure CC=mpicc

################################################################################
Copyright (c) 2011-2013 Los Alamos National Security, LLC.
                        All rights reserved.

This program was prepared by Los Alamos National Security, LLC at Los Alamos
National Laboratory (LANL) under contract No. DE-AC52-06NA25396 with the U.S.
Department of Energy (DOE). All rights in the program are reserved by the DOE
and Los Alamos National Security, LLC. Permission is granted to the public to
copy and use this software without charge, provided that this Notice and any
statement of authorship are reproduced on all copies. Neither the U.S.
Government nor LANS makes any warranty, express or implied, or assumes any
liability or responsibility for the use of this software.