1. Install Spack and setup the environment

Install the latest version of Spack from the official github repo
$ git clone https://github.com/spack/spack.git
Setup the environment
$ source ~/spack/share/spack/setup-env.sh

2. Installing MVAPICH2

MVAPICH2 can be installed from source using Spack.

Use the spack info command to find the variants available to install.
$ spack info mvapich2

Please Note: If you have verbs (either from OFED or MOFED), PSM or PSM2 installed on the system already, you may need to setup external packages in the package.yaml file for rdma-core, psm or opa-psm2, depending on your chosen fabrics.

This is recommended to avoid unexpected runtime failures due to conflicts between the version of the fabric drivers that spack installs and the fabric version available on the system.

# ~/.spack/linux/packages.yaml
# Paste the following so that the dependency is marked as external and is not installed by spack
# Edit the value of spec to be rdma-core/psm/opa-psm2 depending on the dependency of you chosen fabrics.

packages:
  rdma-core:
    buildable: False
    externals:
    - spec: rdma-core
      prefix: /usr
Install the required version of MVAPICH2 based on the variants you desire.
$ spack install mvapich2@2.3.5 %gcc@4.8.5

You can now test your installation by having Spack load it or by navigating to the install directory.

Using Spack to load the installed binaries
$ spack find -l -v -p mvapich2
suqign2 mvapich2@2.3.5~alloca~cuda~debug+regcache+wrapperrpath ch3_rank_bits=32 fabrics=mrail file_systems=auto process_managers=auto threads=multiple  /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.5-suqign2xxq63oao52wtgu4kohkpmhte6

# Note the hash of the required version
$ spack load /suqign2
$ which mpirun_rsh 
$ ~ /spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.5-suqign2dc5uwyir7oaqqtwobrlwabz7k/bin/mpirun_rsh

3. Install XPMEM (If required)

To obtain the best performance from MVAPICH2-X, installation and loading of the XPMEM module is strongly recommended

You can skip this step if you do not intend to use a variant of MVAPICH2-X which uses XPMEM (MVAPICH2-X BASIC-XPMEM, MVAPICH2-X ADVANCED-XPMEM) or if XPMEM is already installed and loaded on your system.

Download the XPMEM module from the following Gitlab link
$ git clone https://gitlab.com/hjelmn/xpmem.git
Build XPMEM
$ cd xpmem
$ ./autogen.sh
$ ./configure --prefix=<>
$ sudo make -j8 install
Load XPMEM
$ sudo insmod /opt/xpmem/lib/modules/4.14.123-111.109.amzn2.x86_64/xpmem.ko
$ sudo chmod 666 /dev/xpmem
You can check if XPMEM is loaded by following command and output:
$ lsmod | grep xpmem
xpmem                  32569  0

4. Install MVAPICH2-X or MVAPICH2-GDR

MVAPICH2-X and MVAPICH2-GDR can only be installed from their respective binary mirrors.

Follow the below steps to setup the mirrors and trust the gpg keys.

Check if the gcc 4.8.5 compiler is loaded and discovered by spack. If not, load the compiler and run the commands below again.
$ spack compiler find
$ spack compiler list
Add the required spack mirrors for MVAPICH2-X and MVAPICH2-GDR
$ spack mirror add mvapich2x http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x
$ spack mirror add mvapich2-gdr http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2-gdr
Trust the public key used to sign the packages
$ wget http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x/build_cache/public.key
$ spack gpg trust public.key
List the available variants and choose the desired spec to install.
$ spack buildcache list -L -v -a
==> 80 cached builds.
-- linux-centos7-x86_64 / gcc@4.8.5 -----------------------------
ru6d56jm7q4jsqjpt2rjilj47pcqbmjt mvapich2x@2.3 distribution=mofed4.5 feature=advanced pmi_version=pmi1 process_managers=mpirun
.
.
.


# Note: ru6d56jm7q4jsqjpt2rjilj47pcqbmjt is the hash of this binary
For MVAPICH2-X, follow the below instructions to install the required binary and resolve any installation errors.
1. First install the dependencies of your chosen MVAPICH2-X varaints. Don't forget to append target=(x86_64/ppc64le).
$ spack install --only dependencies mvapich2x@2.3%gcc@4.8.5 distribution=mofed4.7 feature=advanced-xpmem pmi_version=pmi1 process_managers=mpirun target=x86_64

2. Now get the hash of the binary that you noted earlier and use this hash to install the correct binary. (You might run into some errors which will be addressed below)
login4.frontera(1009)$ spack buildcache install -o /phomjljmbfdy4t5wh4tf7amxkl7r53ss     # (Use -u switch f facing problems with the gpg signatures)
==> buildcache spec(s) matching /phomjljmbfdy4t5wh4tf7amxkl7r53ss

==> Installing buildcache for dependency spec libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: Package for spec libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: Package for spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64
==> Warning: Package for spec xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Warning: Package for spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 already installed.
==> Warning: Package for spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec pmix@3.1.3%gcc@4.8.5+pmi_backwards_compatibility~restful arch=linux-centos7-x86_64 ^hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: Package for spec libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64
==> Warning: Package for spec libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64
==> Warning: Package for spec xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 already installed.
==> Installing buildcache for dependency spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Warning: Package for spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 already installed.
==> Warning: Package for spec libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 already installed.
==> Warning: download_tarball() was unable to download hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 from any configured mirrors
==> Error: Download of binary cache file for spec hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 failed.

3. If the command works without an error, your installation is successful. But, as you can see, you might run into a dependency error.

4. If this happens, you would need to get the full spec from the error of the missing dependency and install it. Then retry installation.

5. Repeat this for all missing dependencies

6. For example, from the below warning message, we can find the full spec of the missing dependency and install the same.
==> Warning: download_tarball() was unable to download hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 from any configured mirrors
$ spack install hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64

7. After installing all missing dependencies, the buildcache install command will succeed
$ login4.frontera(1009)$ spack buildcache install -o /phomjljmbfdy4t5wh4tf7amxkl7r53ss

For MVAPICH2-GDR, follow the below instructions to install the required binary and resolve any installation errors.

Currently supported CUDA versions are below and you can find the correct hash of the binaries further down in this page:-

^cuda@10.1.243

^cuda@10.2.89 (ppc64le only)

^cuda@11.1.1

1. First install the dependencies of your chosen MVAPICH2-GDR varaints. Don't forget to append target=(x86_64/ppc64le).
   Add the correct cuda dependency or configure your packages.yaml with external packages to use system installed cuda. (Refer to spack docs)
$ spack install --only dependencies mvapich2-gdr@2.3.5%gcc@4.8.5 distribution=mofed4.6 pmi_version=pmix process_managers=slurm target=x86_64 ^cuda@10.1.243%gcc@4.8.5 target=x86_64

2. Now get the hash of the binary that you want to install. Refer further below for the binary hash.Use this hash to install the correct binary. (You might run into some errors which will be addressed below)
login4.frontera(1009)$ spack buildcache install -o /xjvq6zjgnhmseorq2tcb3ayoizvfuidl     # (Use -u switch f facing problems with the gpg signatures)

3. If the installation proceeds smoothly, you can skip the next steps.

4. If you run into dependency errors, it is likely that certain dependencies were updated by their maintainers after the binaries were built.
   In these scenarios, we must install the missing dependencies by analyzing the error and warning messages for the full spec of the missing dependencies.

[senthilkumar.16@head spack]$ spack buildcache install -o /xjvq6zjgnhmseorq2tcb3ayoizvfuidl
==> buildcache spec(s) matching /xjvq6zjgnhmseorq2tcb3ayoizvfuidl

==> Installing buildcache for dependency spec pmix@3.1.3%gcc@4.8.5+pmi_backwards_compatibility~restful arch=linux-centos7-x86_64 ^hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Installing buildcache for dependency spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
==> Warning: Package for spec zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 already installed.
==> Warning: download_tarball() was unable to download openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64 from any configured mirrors
==> Error: Download of binary cache file for spec openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 failed.

5. From line below we can see the missing spec. Install that by using the $ spack install command.
==> Installing buildcache for dependency spec pmix@3.1.3%gcc@4.8.5+pmi_backwards_compatibility~restful arch=linux-centos7-x86_64 ^hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64
$ spack install pmix@3.1.3%gcc@4.8.5+pmi_backwards_compatibility~restful arch=linux-centos7-x86_64 ^hwloc@2.2.0%gcc@4.8.5~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-centos7-x86_64 ^libevent@2.1.8%gcc@4.8.5+openssl arch=linux-centos7-x86_64 ^libiconv@1.16%gcc@4.8.5 arch=linux-centos7-x86_64 ^libpciaccess@0.13.5%gcc@4.8.5 arch=linux-centos7-x86_64 ^libxml2@2.9.10%gcc@4.8.5~python arch=linux-centos7-x86_64 ^openssl@1.1.1g%gcc@4.8.5+systemcerts arch=linux-centos7-x86_64 ^xz@5.2.5%gcc@4.8.5~pic arch=linux-centos7-x86_64 ^zlib@1.2.11%gcc@4.8.5+optimize+pic+shared arch=linux-centos7-x86_64

6. Follow this by the buildcache install command and keep repeating the steps untill all dependencies are installed
[senthilkumar.16@head spack]$ spack buildcache install -o /xjvq6zjgnhmseorq2tcb3ayoizvfuidl

You can now test your installation by having spack load it or by navigating to the install directory.

Using spack to load the installed binaries
$ spack find -l -v -p mvapich2-gdr
-- linux-rhel7-ppc64le / gcc@4.8.5 ------------------------------
mkquayp mvapich2-gdr@2.3.5~core_direct+mcast~openacc distribution=mofed4.5 pmi_version=pmi1 process_managers=mpirun /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.5-mkquaypgqutewy4yjrsx4bz7zphwz7bb

# Note the hash of the required version - It's the first word of the previous command's output
$ spack load /mkquayp
$ which mpirun_rsh 

5. Example: How to Run OSU Micro-benchmarks

OMB is installed as default in MVAPICH2-X and MVAPICH2-GDR install path. You can find OMB in ./libexec directory of the install path

Navigate to the install path.

If you did not use spack load command, you may need to prepend MVAPICH2-X library to LD_LIBRARY_PATH like this:

$ export LD_LIBRARY_PATH=$HOME/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.5-mkquaypgqutewy4yjrsx4bz7zphwz7bb/lib/:$LD_LIBRARY_PATH

Run OMB with this command:

$ ./bin/mpirun_rsh -np 2 -hostfile ~/hostfile ./libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

List of Available Spack Binaries

MVAPICH2-X

x86_64 Binaries
OS - Compiler Version MOFED Launcher Type
CentOS7 - GCC 4.8.5 MVAPICH2-X 2.3 MOFED 4.5
MOFED 4.7
MOFED 5.0
mpirun BASIC
BASIC-XPMEM
ADVANCED
ADVANCED-XPMEM
slurm - pmi1, pmi2, pmix BASIC
BASIC-XPMEM
ADVANCED
ADVANCED-XPMEM

MVAPICH2-GDR

x86_64 Binaries
OS - Compiler Version MOFED Launcher Type
CentOS7 - GCC 8.3.0 MVAPICH2-GDR 2.3.5 MOFED 4.7
MOFED 5.0
mpirun ROCm with HIP 3.9.0
CUDA 11.1.1
CUDA 10.1.243
slurm - pmi1, pmi2 ROCm with HIP 3.9.0
CUDA 11.1.1
CUDA 10.1.243
CentOS7 - GCC 4.8.5 MOFED 4.7
MOFED 5.0
mpirun ROCm with HIP 3.9.0
CUDA 11.1.1
CUDA 10.1.243
slurm - pmi1, pmi2 ROCm with HIP 3.9.0
CUDA 11.1.1
CUDA 10.1.243
ppc64le Binaries
OS - Compiler Version MOFED Launcher Type
RHEL7 - GCC 4.8.5 MVAPICH2-GDR 2.3.5 MOFED 4.5 mpirun CUDA 10.2.89
CUDA 10.1.243
List of binary variants and their hashes
OS - Compiler Cuda Version Hash & Spec
CentOS7 - GCC 8.3.0 Cuda 11.1.1 uoi464xymoihgt6sacm7ob4s5hd2v6or mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=mpirun target=x86_64
fqhyfm7mcycwo4qmjomrfcsdaeafowlc mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=slurm target=x86_64
dcqhcw4i6pjvl4y6i5jir7nby2kirkzk mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi2 process_managers=slurm target=x86_64
kwq5xx3r5uch7jaenugb3v7ds3lmyt3z mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=mpirun target=x86_64
dmwvpasoq3fiyd7f3xw2maun632wp2p4 mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=slurm target=x86_64
pgc3tfjrucjfnz4fseflxk765cmdiueg mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi2 process_managers=slurm target=x86_64
gtlwr6gpkju6rp6on4t4umn7kh5kfxws mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmix process_managers= target=x86_64
Cuda 10.1.243 njbjskwnmc6kdtjqdnpsucfpuxpaflpf mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=mpirun target=x86_64
fimct5q4umcjazh5ikkc5liqgi5jb2od mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=slurm target=x86_64
6ay3q75yir4ec273wsv5nip375hbocfm mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi2 process_managers=slurm target=x86_64
cgjcfjfjsz4diikpakg3slowzfdesbux mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=mpirun target=x86_64
yaf2fmhejakpfeeaml7csdprmkwadhql mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=slurm target=x86_64
aejceox5bynehyjaqubqyuztkddp65lt mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi2 process_managers=slurm target=x86_64
ROCm with HIP 3.9.0 35tqsixioeqzghvuwjj6xiivuc5abmpq mvapich2-gdr@2.3.5~core_direct~cuda+mcast~openacc+rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=mpirun target=x86_64
jlg7oknebvq3zpru3khlwgayslziqsh7 mvapich2-gdr@2.3.5~core_direct~cuda+mcast~openacc+rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=slurm target=x86_64
esi7ygfeaj6ffz3l2hok5jpbynqmmrg6 mvapich2-gdr@2.3.5~core_direct~cuda+mcast~openacc+rocm distribution=mofed4.7 pmi_version=pmi2 process_managers=slurm target=x86_64
fa7qcs4ilrxkyb7c37sxrurtkgdjjbdd mvapich2-gdr@2.3.5~core_direct~cuda+mcast~openacc+rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=mpirun target=x86_64
v7slbp2xoaekfibaz6v7ufhhpidx2fmz mvapich2-gdr@2.3.5~core_direct~cuda+mcast~openacc+rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=slurm target=x86_64
nktnnz7woutxgv2aj6fcdkhv65lvtwtk mvapich2-gdr@2.3.5~core_direct~cuda+mcast~openacc+rocm distribution=mofed5.0 pmi_version=pmi2 process_managers=slurm target=x86_64
CentOS7 - GCC 4.8.5 Cuda 11.1.1 tbg6n2ztcdfgvmvr7zjlkky3s7qnm5gh mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=mpirun target=x86_64
wc6ocbafygnkffzjgjteyc3i6vu3ys3s mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=slurm target=x86_64
36nve4yhasbrfedhgzrz4oka4ep4quk5 mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi2 process_managers=slurm target=x86_64
y3bfvtblg5ac5ihu75o4zb773mqb4d6v mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=mpirun target=x86_64
s7nntgxberkoiaaegrggy3wvsgotya3z mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=slurm target=x86_64
dtqs6qsvumyelron3rstiod7jdjmdqr7 mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi2 process_managers=slurm target=x86_64
ijjeson76jdrc2vfxbmisaajnjaunuqu mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmix process_managers=slurm target=x86_64
Cuda 10.1.243 b3bte7jq453k6keabotylbvz7hdefgpo mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=mpirun target=x86_64
dallxrrl3onbbnchti6ukep4zh2zpu3s mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi1 process_managers=slurm target=x86_64
xsmsec5orrbq37etinjxyn7lyjs3bh4t mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmi2 process_managers=slurm target=x86_64
uef67x7jxr7hk4xpei3jalexa2ksib5w mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=mpirun target=x86_64
mnxmojyb5h6byrggzorlaqfwqpzhcjgu mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi1 process_managers=slurm target=x86_64
dsbvoyj32dh5zcwbjpxl5h4yfv7qhopq mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed5.0 pmi_version=pmi2 process_managers=slurm target=x86_64
RHEL7-PPC - GCC 4.8.5 Cuda 10.2.89 7kaljq2dtftu4tbwmcz7d4sfe2rcbtvo mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.5 pmi_version=pmi1 process_managers=mpirun
Cuda 10.1.243 3e4gk7qp4nv6jh63c77uwwpyjlqphhxo mvapich2-gdr@2.3.5~core_direct+cuda+mcast~openacc~rocm distribution=mofed4.5 pmi_version=pmi1 process_managers=mpirun