1. Install Spack and setup the environment

Install the latest version of Spack from the official github repo
$ git clone https://github.com/spack/spack.git
# It is recommended to use v0.16.2 to avoid issues with buildcache index formatting differences between versions
$ git checkout v0.16.2
          
Setup the environment
$ source ~/spack/share/spack/setup-env.sh

2. Installing MVAPICH2

MVAPICH2 can be installed from source using Spack.

Use the spack info command to find the variants available to install.
$ spack info mvapich2

Please Note: If you have verbs (either from OFED or MOFED), PSM or PSM2 installed on the system already, you may need to setup external packages in the package.yaml file for rdma-core, psm or opa-psm2, depending on your chosen fabrics.

This is recommended to avoid unexpected runtime failures due to conflicts between the version of the fabric drivers that spack installs and the fabric version available on the system.

# ~/.spack/linux/packages.yaml
# Paste the following so that the dependency is marked as external and is not installed by spack
# Edit the value of spec to be rdma-core/psm/opa-psm2 depending on the dependency of you chosen fabrics.

packages:
  rdma-core:
    buildable: False
    externals:
    - spec: rdma-core
      prefix: /usr
Install the required version of MVAPICH2 based on the variants you desire.
$ spack install mvapich2@2.3.6 %gcc@4.8.5

You can now test your installation by having Spack load it or by navigating to the install directory.

Using Spack to load the installed binaries
$ spack find -l -v -p mvapich2
suqign2 mvapich2@2.3.6~alloca~cuda~debug+regcache+wrapperrpath ch3_rank_bits=32 fabrics=mrail file_systems=auto process_managers=auto threads=multiple  /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.6-suqign2xxq63oao52wtgu4kohkpmhte6

# Note the hash of the required version
$ spack load /suqign2
$ which mpirun_rsh 
$ ~ /spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.5-suqign2dc5uwyir7oaqqtwobrlwabz7k/bin/mpirun_rsh
              

3. Install XPMEM (If required)

To obtain the best performance from MVAPICH2-X, installation and loading of the XPMEM module is strongly recommended

You can skip this step if you do not intend to use a variant of MVAPICH2-X which uses XPMEM (MVAPICH2-X BASIC-XPMEM, MVAPICH2-X ADVANCED-XPMEM) or if XPMEM is already installed and loaded on your system.

Download the XPMEM module from the following Gitlab link
$ git clone https://gitlab.com/hjelmn/xpmem.git
Build XPMEM
$ cd xpmem
$ ./autogen.sh
$ ./configure --prefix=<>
$ sudo make -j8 install
Load XPMEM
$ sudo insmod /opt/xpmem/lib/modules/4.14.123-111.109.amzn2.x86_64/xpmem.ko
$ sudo chmod 666 /dev/xpmem
You can check if XPMEM is loaded by following command and output:
$ lsmod | grep xpmem
xpmem                  32569  0

4. Install MVAPICH2-X or MVAPICH2-GDR

MVAPICH2-X and MVAPICH2-GDR can only be installed from their respective binary mirrors.

Follow the below steps to setup the mirrors and trust the gpg keys.

Check if the gcc 4.8.5 compiler is loaded and discovered by spack. If not, load the compiler and run the commands below again.
$ spack compiler find
$ spack compiler list
Add the required spack mirrors for MVAPICH2-X and MVAPICH2-GDR
$ spack mirror add mvapich2x http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x
$ spack mirror add mvapich2-gdr http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2-gdr
Trust the public key used to sign the packages
$ wget http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x/build_cache/public.key
$ spack gpg trust public.key
List the available variants and choose the desired spec to install.
$ spack buildcache list -L -v -a
==> 80 cached builds.
-- linux-centos7-x86_64 / gcc@4.8.5 -----------------------------
ru6d56jm7q4jsqjpt2rjilj47pcqbmjt mvapich2x@2.3 distribution=mofed4.5 feature=advanced pmi_version=pmi1 process_managers=mpirun
.
.
.


# Note: ru6d56jm7q4jsqjpt2rjilj47pcqbmjt is the hash of this binary 
If you do not see any cached build entries, verify that you mirror is set up correctly using spack mirror list. Also, check your spack version. The buildcache is currently generated with spack v0.16.2, after which it is possible that there are changes to buildcache index format, making it impossible for later clients to read. If your version of spack is a later version or a non-release version (it will have more characters after v0.16.2), it is recommended you switch to version v0.16.2. With your working directory set as the spack repository root: git checkout v0.16.2 If you are unable to find a cache entry, follow the directions for requesting spack buildcache entries below.
For MVAPICH2-X, follow the below instructions to install the required binary and resolve any installation errors.
1. First install the dependencies of your chosen MVAPICH2-X varaints. Don't forget to append target=(x86_64/ppc64le).
$ spack install --only dependencies mvapich2x@2.3%gcc@4.8.5 distribution=mofed4.7 feature=advanced-xpmem pmi_version=pmi1 process_managers=mpirun target=x86_64

2. Now get the hash of the binary that you noted earlier and use this hash to install the correct binary. (You might run into some errors which will be addressed below)
login4.frontera(1009)$ spack buildcache install -o /phomjljmbfdy4t5wh4tf7amxkl7r53ss     # (Use -u switch f facing problems with the gpg signatures)
==> buildcache spec(s) matching /phomjljmbfdy4t5wh4tf7amxkl7r53ss

==> Installing buildcache for dependency spec libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: Package for spec libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: Package for spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64
==> Warning: Package for spec xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Warning: Package for spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 already installed.
==> Warning: Package for spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec pmix@3.1.3%gcc@4.8.5+pmi_backwards_compatibility~restful arch=linux-centos7-x86_64 ^hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: Package for spec libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: Package for spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64
==> Warning: Package for spec xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Warning: Package for spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 already installed.
==> Warning: Package for spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 already installed.
==> Warning: download_tarball() was unable to download hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 from any configured mirrors
==> Error: Download of binary cache file for spec hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 failed.

3. If the command works without an error, your installation is successful. But, as you can see, you might run into a dependency error.

4. If this happens, you would need to get the full spec from the error of the missing dependency and install it. Then retry installation.

5. Repeat this for all missing dependencies

6. For example, from the below warning message, we can find the full spec of the missing dependency and install the same.
==> Warning: download_tarball() was unable to download hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 from any configured mirrors
$ spack install hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64

7. After installing all missing dependencies, the buildcache install command will succeed
$ login4.frontera(1009)$ spack buildcache install -o /phomjljmbfdy4t5wh4tf7amxkl7r53ss
For MVAPICH2-GDR, follow the below instructions to install the required binary and resolve any installation errors.

Currently supported CUDA versions are below and you can find the correct hash of the binaries further down in this page:-

^cuda@10.1.243

^cuda@10.2.89 (ppc64le only)

^cuda@11.1.1

1. First install the dependencies of your chosen MVAPICH2-GDR varaints. Don't forget to append target=(x86_64/ppc64le).
   Add the correct cuda dependency or configure your packages.yaml with external packages to use system installed cuda. (Refer to spack docs)
$ spack install --only dependencies mvapich2-gdr@2.3.5%gcc@4.8.5 distribution=mofed4.6 pmi_version=pmix process_managers=slurm target=x86_64 ^cuda@10.1.243%gcc@4.8.5 target=x86_64

2. Now get the hash of the binary that you want to install. Refer further below for the binary hash.Use this hash to install the correct binary. (You might run into some errors which will be addressed below)
login4.frontera(1009)$ spack buildcache install -o /xjvq6zjgnhmseorq2tcb3ayoizvfuidl     # (Use -u switch f facing problems with the gpg signatures)

3. If the installation proceeds smoothly, you can skip the next steps.

4. If you run into dependency errors, it is likely that certain dependencies were updated by their maintainers after the binaries were built.
   In these scenarios, we must install the missing dependencies by analyzing the error and warning messages for the full spec of the missing dependencies.

[senthilkumar.16@head spack]$ spack buildcache install -o /xjvq6zjgnhmseorq2tcb3ayoizvfuidl
==> buildcache spec(s) matching /xjvq6zjgnhmseorq2tcb3ayoizvfuidl

==> Installing buildcache for dependency spec pmix@3.1.3%gcc@4.8.5+pmi_backwards_compatibility~restful arch=linux-centos7-x86_64 ^hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Warning: Package for spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 already installed.
==> Warning: download_tarball() was unable to download openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 from any configured mirrors
==> Error: Download of binary cache file for spec openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 failed.

5. From line below we can see the missing spec. Install that by using the $ spack install command.
==> Installing buildcache for dependency spec pmix@3.1.3%gcc@4.8.5+pmi_backwards_compatibility~restful arch=linux-centos7-x86_64 ^hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
$ spack install pmix@3.1.3%gcc@4.8.5+pmi_backwards_compatibility~restful arch=linux-centos7-x86_64 ^hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64

6. Follow this by the buildcache install command and keep repeating the steps untill all dependencies are installed
[senthilkumar.16@head spack]$ spack buildcache install -o /xjvq6zjgnhmseorq2tcb3ayoizvfuidl

You can now test your installation by having spack load it or by navigating to the install directory.

Using spack to load the installed binaries
$ spack find -l -v -p mvapich2-gdr
-- linux-rhel7-ppc64le / gcc@4.8.5 ------------------------------
mkquayp mvapich2-gdr@2.3.5~core_direct+mcast~openacc distribution=mofed4.5 pmi_version=pmi1 process_managers=mpirun /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.5-mkquaypgqutewy4yjrsx4bz7zphwz7bb

# Note the hash of the required version - It's the first word of the previous command's output
$ spack load /mkquayp
$ which mpirun_rsh

5. Example: How to Run OSU Micro-benchmarks

OMB is installed as default in MVAPICH2-X and MVAPICH2-GDR install path. You can find OMB in ./libexec directory of the install path

Navigate to the install path.

If you did not use spack load command, you may need to prepend MVAPICH2-X library to LD_LIBRARY_PATH like this:

$ export LD_LIBRARY_PATH=$HOME/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.5-mkquaypgqutewy4yjrsx4bz7zphwz7bb/lib/:$LD_LIBRARY_PATH

Run OMB with this command:

$ ./bin/mpirun_rsh -np 2 -hostfile ~/hostfile ./libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

Requesting what generation of buildcache entries

To request generation of a buildcache entry with a specific configuration, please email mvapich-help@cse.ohio-state.edu with what you would like the entry to be built with. Here is a list of dependencies:
  • Operating System (RHEL7, CentOS7)
  • Compiler (Version of gcc)
  • Launcher (slurm, mpirun)
  • OFED Version
  • Architecture
  • Cuda/ROCm Version (for GDR only)
  • with or without mcast, openacc, and/or core_direct (for GDR only)
  • Basic, Basic-XPMEM, Advanced, or Advanced-XPMEM (for X only)