1. Install Spack and setup the environment

Install the latest version of Spack from the official github repo
$ git clone https://github.com/spack/spack.git
# It is recommended to use at least v0.18.0 to avoid issues with buildcache index formatting differences between earlier versions
$ git checkout v0.18.0
# Since the package specs are part of the spack repo itself, users might want
# to checkout the latest version of the spec
# This step is needed to use the latest version of mvapich2-gdr if the latest spack release doesn't have the latest package configuration
$ git checkout develop -- var/spack/repos/builtin/packages/mvapich2-gdr/package.py
$ git checkout develop -- var/spack/repos/builtin/packages/mvapich2x/package.py
          
Setup the environment
$ source ~/spack/share/spack/setup-env.sh

2. Installing MVAPICH2

MVAPICH2 can be installed from source using Spack.

Use the spack info command to find the variants available to install.
$ spack info mvapich2

Please Note: If you have verbs (either from OFED or MOFED), PSM or PSM2 installed on the system already, you may need to setup external packages in the package.yaml file for rdma-core, psm or opa-psm2, depending on your chosen fabrics.

This is recommended to avoid unexpected runtime failures due to conflicts between the version of the fabric drivers that spack installs and the fabric version available on the system.

# ~/.spack/linux/packages.yaml
# Paste the following so that the dependency is marked as external and is not installed by spack
# Edit the value of spec to be rdma-core/psm/opa-psm2 depending on the dependency of you chosen fabrics.

packages:
  rdma-core:
    buildable: False
    externals:
    - spec: rdma-core
      prefix: /usr
Install the required version of MVAPICH2 based on the variants you desire.
$ spack install mvapich2@2.3.7 %gcc@4.8.5

You can now test your installation by having Spack load it or by navigating to the install directory.

Using Spack to load the installed binaries
$ spack find -l -v -p mvapich2
suqign2 mvapich2@2.3.7~alloca~cuda~debug+regcache+wrapperrpath ch3_rank_bits=32 fabrics=mrail file_systems=auto process_managers=auto threads=multiple  /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.7-suqign2xxq63oao52wtgu4kohkpmhte6

# Note the hash of the required version
$ spack load /suqign2
$ which mpirun_rsh 
$ ~ /spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.7-suqign2dc5uwyir7oaqqtwobrlwabz7k/bin/mpirun_rsh
              

3. Install XPMEM (If required)

To obtain the best performance from MVAPICH2-X, installation and loading of the XPMEM module is strongly recommended

You can skip this step if you do not intend to use a variant of MVAPICH2-X which uses XPMEM (MVAPICH2-X BASIC-XPMEM, MVAPICH2-X ADVANCED-XPMEM) or if XPMEM is already installed and loaded on your system.

Download the XPMEM module from the following Gitlab link
$ git clone https://gitlab.com/hjelmn/xpmem.git
Build XPMEM
$ cd xpmem
$ ./autogen.sh
$ ./configure --prefix=<>
$ sudo make -j8 install
Load XPMEM
$ sudo insmod /opt/xpmem/lib/modules/4.14.123-111.109.amzn2.x86_64/xpmem.ko
$ sudo chmod 666 /dev/xpmem
You can check if XPMEM is loaded by following command and output:
$ lsmod | grep xpmem
xpmem                  32569  0

4. Install MVAPICH2-X or MVAPICH2-GDR

MVAPICH2-X and MVAPICH2-GDR can only be installed from their respective binary mirrors.

Follow the below steps to setup the mirrors and trust the gpg keys.

Add the required spack mirrors for MVAPICH2-X and MVAPICH2-GDR
$ spack mirror add mvapich2x http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x
$ spack mirror add mvapich2-gdr http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2-gdr
Trust the public key used to sign the packages
$ wget http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x/build_cache/public.key
$ spack gpg trust public.key
List the available variants and choose the desired spec to install.
$ spack buildcache list -L -v -a
==> 80 cached builds.
-- linux-centos7-x86_64 / gcc@4.8.5 -----------------------------
ru6d56jm7q4jsqjpt2rjilj47pcqbmjt mvapich2x@2.3 distribution=mofed4.5 feature=advanced pmi_version=pmi1 process_managers=mpirun
.
.
.


# Note: ru6d56jm7q4jsqjpt2rjilj47pcqbmjt is the hash of this binary 
If you do not see any cached build entries, verify that you mirror is set up correctly using spack mirror list. Also, check your spack version. The buildcache is currently generated with spack v0.18.0, before which it is possible that there are differences in the buildcache index format, making it impossible for earlier clients to read. If your version of spack is an earlier version or a non-release version (it will have more characters after v0.18.0), it is recommended you switch to version v0.18.0. With your working directory set as the spack repository root: git checkout v0.18.0 If you are unable to find a cache entry built with your desired configuration, follow the directions for requesting spack buildcache entries below.
Check if the version of gcc corresponding to the cache entry you want to install is known by spack.
 $ spack compiler list
# if the compiler version needed is not listed, use "spack find" to add it

# this command will search the PATH variable for compilers
$ spack compiler find

# it might be easier to explictly specify the path the gcc executable
$ spack compiler find /path/to/gcc
For MVAPICH2-GDR and MVAPICH2-X, follow the below instructions to install the required binary and resolve any installation errors.

GDR only: Currently supported CUDA versions are below:

^cuda@11.7.0

^cuda@11.6.2

^cuda@11.5.1

^cuda@11.4.2

^cuda@11.3.1

^cuda@11.2.2

^cuda@11.1.1

^cuda@11.0.3

^cuda@10.2.89

1. First install the dependencies of your chosen MVAPICH2-GDR variants. Don't forget to append target=(x86_64/ppc64le).
   Add the correct cuda dependency or configure your packages.yaml with external packages to use system installed cuda. (Refer to spack docs)
$ spack install --only dependencies mvapich2-gdr@2.3.7%gcc@4.8.5 distribution=mofed4.7 pmi_version=pmix process_managers=slurm target=x86_64 ^cuda@11.1.0%gcc@4.8.5 target=x86_64

2. Now get the hash of the binary that you want to install. Refer further below for the binary hash.Use this hash to install the correct binary. (You might run into some errors which will be addressed below)
login4.frontera(1009)$ spack buildcache install -o /44aoflbqebw3xbgp4rbncvrjwbjcwhs4     # (Use -u switch if facing problems with the gpg signatures)

3. If the installation proceeds smoothly, you can skip the next steps.

4. If you run into dependency errors, it is likely that certain dependencies were updated by their maintainers after the binaries were built.
   In these scenarios, we must install the missing dependencies by analyzing the error and warning messages for the full spec of the missing dependencies.

[senthilkumar.16@head spack]$ spack buildcache install -o /44aoflbqebw3xbgp4rbncvrjwbjcwhs4
==> buildcache spec(s) matching /44aoflbqebw3xbgp4rbncvrjwbjcwhs4 

==> Installing buildcache for dependency spec cuda@11.1.0%gcc@4.8.5 arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: download_tarball() was unable to download libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 from any configured mirrors
==> Error: Download of binary cache file for spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 failed.

5. From line above we can see the missing spec. Install that by using the $ spack install command. You may want to add --fresh to the command if installing from build cache complains about a dependency that you just installed.
[senthilkumar.16@head spack]$ spack install libiconv@1.16%gcc@4.8.5
==> Installing libiconv-1.16-qazxaa4lnd2ukd44fjzdhayyxvcg3wyu
==> No binary for libiconv-1.16-qazxaa4lnd2ukd44fjzdhayyxvcg3wyu found: installing from source
==> Using cached archive: /home/contini.26/spack-related/spack/var/spack/cache/_source-cache/archive/e6/e6a1b1b589654277ee790cce3734f07876ac4ccfaecbee8afa0b649cf529cc04.tar.gz
==> libiconv: Executing phase: 'autoreconf'
==> libiconv: Executing phase: 'configure'
==> libiconv: Executing phase: 'build'
==> libiconv: Executing phase: 'install'
[+] /home/contini.26/spack-related/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/libiconv-1.16-qazxaa4lnd2ukd44fjzdhayyxvcg3wyu

6. Follow this by the buildcache install command and keep repeating the steps untill all dependencies are installed
[senthilkumar.16@head spack]$ spack buildcache install -o /44aoflbqebw3xbgp4rbncvrjwbjcwhs4

You can now test your installation by having spack load it or by navigating to the install directory.

Using spack to load the installed binaries
$ spack find -l -v -p mvapich2-gdr
-- linux-centos7-x86_64 / gcc@4.8.5 -----------------------------
44aoflb mvapich2-gdr@2.3.7+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmix process_managers=slurm  /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.7-44aoflbqebw3xbgp4rbncvrjwbjcwhs4

# Note the hash of the required version - It's the first word of the previous command's output
$ spack load /44aoflb
$ which mpirun_rsh

5. Example: How to Run OSU Micro-benchmarks

OMB is installed as default in MVAPICH2-X and MVAPICH2-GDR install path. You can find OMB in ./libexec directory of the install path

Navigate to the install path.

If you did not use spack load command, you may need to prepend MVAPICH2-X/GDR library to LD_LIBRARY_PATH like this:

$ export LD_LIBRARY_PATH=$HOME/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.7-44aoflbqebw3xbgp4rbncvrjwbjcwhs4/lib/:$LD_LIBRARY_PATH

Run OMB with this command:

$ ./bin/mpirun_rsh -np 2 -hostfile ~/hostfile ./libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

Requesting generation of buildcache entries

To request generation of a buildcache entry with a specific configuration, please email mvapich-help@cse.ohio-state.edu with what you would like the entry to be built with. Here is a list of dependencies:
  • Operating System (RHEL7, CentOS7)
  • Compiler (Version of gcc)
  • Launcher (slurm, mpirun)
  • OFED Version
  • Architecture
  • Cuda/ROCm Version (for GDR only)
  • with or without mcast and/or openacc (for GDR only)
  • Basic, Basic-XPMEM, Advanced, or Advanced-XPMEM (for X only)