1. Install Spack and setup the environment

Install the latest version of Spack from the official github repo
$ git clone https://github.com/spack/spack.git
# It is recommended to use at least version 0.20.0 to avoid errors caused by older versions
$ git checkout v0.20.0
# Since the package specs are part of the spack repo itself, users might want
# to checkout the latest version of the spec
# This step is needed to use the latest version of mvapich2-gdr if the latest spack release doesn't have the latest package configuration
$ git checkout develop -- var/spack/repos/builtin/packages/mvapich/package.py
Setup the environment
$ source ~/spack/share/spack/setup-env.sh

2. Installing MVAPICH

Starting from Version 3 onward MVAPICH2 will now become MVAPICH. A new spack package will also be used. The old MVAPICH2 package will still be availible but not updated.

MVAPICH can be installed from source using Spack.

Use the spack info command to find the variants available to install.
$ spack info mvapich

Please Note: If you have verbs (either from OFED or MOFED), PSM or PSM2 installed on the system already, you may need to setup external packages in the package.yaml file for rdma-core, psm or opa-psm2, depending on your chosen fabrics.

This is recommended to avoid unexpected runtime failures due to conflicts between the version of the fabric drivers that spack installs and the fabric version available on the system.

# ~/.spack/linux/packages.yaml
# Paste the following so that the dependency is marked as external and is not installed by spack
# Edit the value of spec to be rdma-core/psm/opa-psm2 depending on the dependency of you chosen fabrics.

    buildable: False
    - spec: rdma-core
      prefix: /usr
Install the required version of MVAPICH based on the variants you desire.
$ spack install mvapich@3.0b %gcc@8.5.0

You can now test your installation by having Spack load it or by navigating to the install directory.

Using Spack to load the installed binaries
$ spack find -l -v -p mvapich
3dr7n64 mvapich@3.0b~alloca~cuda~debug+regcache+wrapperrpath build_system=autotools ch3_rank_bits=32 file_systems=auto netmod=ofi pmi_version=simple process_managers=auto threads=multiple  /home/user/spack/opt/spack/linux-rocky8-zen/gcc-8.5.0/mvapich-3.0b-3dr7n642u7y6aoeoglerqihrkkbcuwed

# Note the hash of the required version
$ spack load /3dr7n64
$ which mpirun_rsh 
$ ~/spack/opt/spack/linux-rocky8-zen/gcc-8.5.0/mvapich-3.0b-3dr7n642u7y6aoeoglerqihrkkbcuwed/bin/mpirun_rsh