1. Install Spack and setup the environment

Install the latest version of Spack from the official github repo
$ git clone https://github.com/spack/spack.git
Setup the environment
$ source ~/spack/share/spack/setup-env.sh

2. Installing MVAPICH2

MVAPICH2 can be installed from source using Spack.

Use the spack info command to find the variants available to install.
$ spack info mvapich2
Install the required version of MVAPICH2 based on the variants you desire.
$ spack install mvapich2@2.3.4 %gcc@8.3.0

You can now test your installation by having Spack load it or by navigating to the install directory.

Using Spack to load the installed binaries
$ spack find -l -v -p mvapich2
suqign2 mvapich2@2.3.4~alloca~cuda~debug+regcache+wrapperrpath ch3_rank_bits=32 fabrics=mrail file_systems=auto process_managers=auto threads=multiple  /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.4-suqign2xxq63oao52wtgu4kohkpmhte6

# Note the hash of the required version
$ spack load /suqign2
$ which mpirun_rsh 
$ ~ /spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2x-2.3rc3- suqign2dc5uwyir7oaqqtwobrlwabz7k/bin/mpirun_rsh

3. Install XPMEM (If required)

To obtain the best performance from MVAPICH2-X, installation and loading of the XPMEM module is strongly recommended

You can skip this step if you do not intend to use a variant of MVAPICH2-X which uses XPMEM (MVAPICH2-X BASIC-XPMEM, MVAPICH2-X ADVANCED-XPMEM) or if XPMEM is already installed and loaded on your system.

Download the XPMEM module from the following Gitlab link
$ git clone https://gitlab.com/hjelmn/xpmem.git
Build XPMEM
$ cd xpmem
$ ./autogen.sh
$ ./configure --prefix=<>
$ sudo make -j8 install
Load XPMEM
$ sudo insmod /opt/xpmem/lib/modules/4.14.123-111.109.amzn2.x86_64/xpmem.ko
$ sudo chmod 666 /dev/xpmem
You can check if XPMEM is loaded by following command and output:
$ lsmod | grep xpmem
xpmem                  32569  0

4. Install MVAPICH2-X or MVAPICH2-GDR

MVAPICH2-X and MVAPICH2-GDR can only be installed from their respective binary mirrors.

Follow the below steps to setup the mirrors and trust the gpg keys.

Check if the gcc 4.8.5 compiler is loaded and discovered by spack. If not, load the compiler and run the commands below again.
$ spack compiler find
$ spack compiler list
Add the required spack mirrors for MVAPICH2-X and MVAPICH2-GDR
$ spack mirror add mvapich2x http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x
$ spack mirror add mvapich2-gdr http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2-gdr
Trust the public key used to sign the packages
$ wget http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x/build_cache/public.key
$ spack gpg trust public.key
List the available variants and choose the desired spec to install.
$ spack buildcache list -L -v -a
For MVAPICH2-X, find the name of the package you wish to install and append the target architecture type(x86_64/ppc64le) to the installation command
$ spack install mvapich2x@2.3 distribution=mofed4.6 feature=advanced-xpmem pmi_version=pmi1 process_managers=mpirun target=x86_64
For MVAPICH2-GDR, append the cuda version dependency followed by the target architecture type to the installation command

Currently supported CUDA versions are:-

^cuda@9.2.88

^cuda@10.1.243

^cuda@10.2.89

$ spack install mvapich2-gdr@2.3.3~core_direct+mcast~openacc distribution=mofed4.5 pmi_version=pmi1 process_managers=mpirun ^cuda@9.2.88 target=x86_64

You can now test your installation by having spack load it or by navigating to the install directory.

Using spack to load the installed binaries
$ spack find -l -v -p mvapich2-gdr
-- linux-rhel7-ppc64le / gcc@4.8.5 ------------------------------
mkquayp mvapich2-gdr@2.3.3~core_direct+mcast~openacc cuda_version=10.2 distribution=mofed4.5 pmi_version=pmi1 process_managers=mpirun /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.3-mkquaypgqutewy4yjrsx4bz7zphwz7bb

# Note the hash of the required version - It's the first word of the previous command's output
$ spack load /mkquayp
$ which mpirun_rsh 

5. Example: How to Run OSU Micro-benchmarks

OMB is installed as default in MVAPICH2-X and MVAPICH2-GDR install path. You can find OMB in ./libexec directory of the install path

Navigate to the install path.

If you did not use spack load command, you may need to prepend MVAPICH2-X library to LD_LIBRARY_PATH like this:

$ export LD_LIBRARY_PATH=$HOME/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.3-mkquaypgqutewy4yjrsx4bz7zphwz7bb/lib/:$LD_LIBRARY_PATH

Run OMB with this command:

$ ./bin/mpirun_rsh -np 2 -hostfile ~/hostfile ./libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

List of Available Spack Binaries

MVAPICH2-X

x86_64 Binaries
Version MOFED Launcher Type
CentOS7 - GCC 4.8.5 MVAPICH2-X 2.3 MOFED 4.5
MOFED 4.6
MOFED 4.7
MOFED 5.0
mpirun BASIC
BASIC-XPMEM
ADVANCED
ADVANCED-XPMEM
slurm - pmi1, pmi2, pmix BASIC
BASIC-XPMEM
ADVANCED
ADVANCED-XPMEM
MVAPICH2-X 2.3rc3 MOFED 4.5
MOFED 4.6
MOFED 4.7
MOFED 5.0
mpirun BASIC
BASIC-XPMEM
ADVANCED
ADVANCED-XPMEM
slurm - pmi1, pmi2, pmix BASIC
BASIC-XPMEM
ADVANCED
ADVANCED-XPMEM

MVAPICH2-GDR

x86_64 Binaries
Version MOFED Launcher Type
CentOS7 - GCC 4.8.5 MVAPICH2-GDR 2.3.4 MOFED 4.5
MOFED 4.6
MOFED 4.7
MOFED 5.0
mpirun CUDA 9.2.88
CUDA 10.1.243
CUDA 10.2.89
slurm - pmi1, pmi2, pmix CUDA 9.2.88
CUDA 10.1.243
CUDA 10.2.89
MVAPICH2-GDR 2.3.3 MOFED 4.5
MOFED 4.6
MOFED 4.7
MOFED 5.0
mpirun CUDA 9.2.88
CUDA 10.1.243
CUDA 10.2.89
slurm - pmi1, pmi2, pmix CUDA 9.2.88
CUDA 10.1.243
CUDA 10.2.89
ppc64le Binaries
Version MOFED Launcher Type
RHEL7 - GCC 4.8.5 MVAPICH2-GDR 2.3.4 MOFED 4.5 mpirun CUDA 10.2.89
slurm - pmix CUDA 10.2.89
MVAPICH2-GDR 2.3.3 MOFED 4.5
mpirun CUDA 10.2.89
slurm - pmix CUDA 10.2.89