1. Install Spack and setup the environment
$ git clone https://github.com/spack/spack.git # It is recommended to use at least v0.18.0 to avoid issues with buildcache index formatting differences between earlier versions $ git checkout v0.18.0 # Since the package specs are part of the spack repo itself, users might want # to checkout the latest version of the spec # This step is needed to use the latest version of mvapich2-gdr if the latest spack release doesn't have the latest package configuration $ git checkout develop -- var/spack/repos/builtin/packages/mvapich2-gdr/package.py $ git checkout develop -- var/spack/repos/builtin/packages/mvapich2x/package.py
$ source ~/spack/share/spack/setup-env.sh
2. Installing MVAPICH2
MVAPICH2 can be installed from source using Spack.
$ spack info mvapich2
Please Note: If you have verbs (either from OFED or MOFED), PSM or PSM2
installed on the system already, you may need to setup external packages in the
package.yaml file for rdma-core, psm or opa-psm2, depending on your chosen fabrics.
This is recommended to avoid unexpected runtime failures due to conflicts between the version of the fabric drivers that spack installs and the fabric version available on the system.
# ~/.spack/linux/packages.yaml # Paste the following so that the dependency is marked as external and is not installed by spack # Edit the value of spec to be rdma-core/psm/opa-psm2 depending on the dependency of you chosen fabrics. packages: rdma-core: buildable: False externals: - spec: rdma-core prefix: /usr
$ spack install firstname.lastname@example.org %email@example.com
You can now test your installation by having Spack load it or by navigating to the install directory.
$ spack find -l -v -p mvapich2 suqign2 firstname.lastname@example.org~alloca~cuda~debug+regcache+wrapperrpath ch3_rank_bits=32 fabrics=mrail file_systems=auto process_managers=auto threads=multiple /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.7-suqign2xxq63oao52wtgu4kohkpmhte6 # Note the hash of the required version $ spack load /suqign2 $ which mpirun_rsh $ ~ /spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-2.3.7-suqign2dc5uwyir7oaqqtwobrlwabz7k/bin/mpirun_rsh
3. Install XPMEM (If required)
To obtain the best performance from MVAPICH2-X, installation and loading of the XPMEM module is strongly recommended
You can skip this step if you do not intend to use a variant of MVAPICH2-X which uses XPMEM (MVAPICH2-X BASIC-XPMEM, MVAPICH2-X ADVANCED-XPMEM) or if XPMEM is already installed and loaded on your system.
$ git clone https://gitlab.com/hjelmn/xpmem.git
$ cd xpmem $ ./autogen.sh $ ./configure --prefix=<> $ sudo make -j8 install
$ sudo insmod /opt/xpmem/lib/modules/4.14.123-111.109.amzn2.x86_64/xpmem.ko $ sudo chmod 666 /dev/xpmem
$ lsmod | grep xpmem xpmem 32569 0
4. Install MVAPICH2-X or MVAPICH2-GDR
MVAPICH2-X and MVAPICH2-GDR can only be installed from their respective binary mirrors.
Follow the below steps to setup the mirrors and trust the gpg keys.
$ spack mirror add mvapich2x http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x $ spack mirror add mvapich2-gdr http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2-gdr
$ wget http://mvapich.cse.ohio-state.edu/download/mvapich/spack-mirror/mvapich2x/build_cache/public.key $ spack gpg trust public.key
$ spack buildcache list -L -v -a ==> 80 cached builds. -- linux-centos7-x86_64 / email@example.com ----------------------------- ru6d56jm7q4jsqjpt2rjilj47pcqbmjt firstname.lastname@example.org distribution=mofed4.5 feature=advanced pmi_version=pmi1 process_managers=mpirun . . . # Note: ru6d56jm7q4jsqjpt2rjilj47pcqbmjt is the hash of this binary
$ spack compiler list # if the compiler version needed is not listed, use "spack find" to add it # this command will search the PATH variable for compilers $ spack compiler find # it might be easier to explictly specify the path the gcc executable $ spack compiler find /path/to/gcc
GDR only: Currently supported CUDA versions are below:
1. First install the dependencies of your chosen MVAPICH2-GDR variants. Don't forget to append target=(x86_64/ppc64le). Add the correct cuda dependency or configure your packages.yaml with external packages to use system installed cuda. (Refer to spack docs) $ spack install --only dependencies email@example.comfirstname.lastname@example.org distribution=mofed4.7 pmi_version=pmix process_managers=slurm target=x86_64 ^email@example.comfirstname.lastname@example.org target=x86_64 2. Now get the hash of the binary that you want to install. Refer further below for the binary hash.Use this hash to install the correct binary. (You might run into some errors which will be addressed below) login4.frontera(1009)$ spack buildcache install -o /44aoflbqebw3xbgp4rbncvrjwbjcwhs4 # (Use -u switch if facing problems with the gpg signatures) 3. If the installation proceeds smoothly, you can skip the next steps. 4. If you run into dependency errors, it is likely that certain dependencies were updated by their maintainers after the binaries were built. In these scenarios, we must install the missing dependencies by analyzing the error and warning messages for the full spec of the missing dependencies. [senthilkumar.16@head spack]$ spack buildcache install -o /44aoflbqebw3xbgp4rbncvrjwbjcwhs4 ==> buildcache spec(s) matching /44aoflbqebw3xbgp4rbncvrjwbjcwhs4 ==> Installing buildcache for dependency spec email@example.comfirstname.lastname@example.org arch=linux-centos7-x86_64 ^email@example.comfirstname.lastname@example.org arch=linux-centos7-x86_64 ^email@example.comfirstname.lastname@example.org~python arch=linux-centos7-x86_64 ^email@example.comfirstname.lastname@example.org~pic arch=linux-centos7-x86_64 ^email@example.comfirstname.lastname@example.org+optimize+pic+shared arch=linux-centos7-x86_64 ==> Installing buildcache for dependency spec email@example.comfirstname.lastname@example.org~python arch=linux-centos7-x86_64 ^email@example.comfirstname.lastname@example.org arch=linux-centos7-x86_64 ^email@example.comfirstname.lastname@example.org~pic arch=linux-centos7-x86_64 ^email@example.comfirstname.lastname@example.org+optimize+pic+shared arch=linux-centos7-x86_64 ==> Installing buildcache for dependency spec email@example.comfirstname.lastname@example.org arch=linux-centos7-x86_64 ==> Warning: download_tarball() was unable to download email@example.comfirstname.lastname@example.org arch=linux-centos7-x86_64 from any configured mirrors ==> Error: Download of binary cache file for spec email@example.comfirstname.lastname@example.org arch=linux-centos7-x86_64 failed. 5. From line above we can see the missing spec. Install that by using the $ spack install command. You may want to add --fresh to the command if installing from build cache complains about a dependency that you just installed. [senthilkumar.16@head spack]$ spack install email@example.comfirstname.lastname@example.org ==> Installing libiconv-1.16-qazxaa4lnd2ukd44fjzdhayyxvcg3wyu ==> No binary for libiconv-1.16-qazxaa4lnd2ukd44fjzdhayyxvcg3wyu found: installing from source ==> Using cached archive: /home/contini.26/spack-related/spack/var/spack/cache/_source-cache/archive/e6/e6a1b1b589654277ee790cce3734f07876ac4ccfaecbee8afa0b649cf529cc04.tar.gz ==> libiconv: Executing phase: 'autoreconf' ==> libiconv: Executing phase: 'configure' ==> libiconv: Executing phase: 'build' ==> libiconv: Executing phase: 'install' [+] /home/contini.26/spack-related/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/libiconv-1.16-qazxaa4lnd2ukd44fjzdhayyxvcg3wyu 6. Follow this by the buildcache install command and keep repeating the steps untill all dependencies are installed [senthilkumar.16@head spack]$ spack buildcache install -o /44aoflbqebw3xbgp4rbncvrjwbjcwhs4
You can now test your installation by having spack load it or by navigating to the install directory.
$ spack find -l -v -p mvapich2-gdr -- linux-centos7-x86_64 / email@example.com ----------------------------- 44aoflb firstname.lastname@example.org+cuda+mcast~openacc~rocm distribution=mofed4.7 pmi_version=pmix process_managers=slurm /home/user/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.7-44aoflbqebw3xbgp4rbncvrjwbjcwhs4 # Note the hash of the required version - It's the first word of the previous command's output $ spack load /44aoflb $ which mpirun_rsh
5. Example: How to Run OSU Micro-benchmarks
OMB is installed as default in MVAPICH2-X and MVAPICH2-GDR install path. You can find OMB in ./libexec directory of the install path
Navigate to the install path.
If you did not use spack load command, you may need to prepend MVAPICH2-X/GDR library to LD_LIBRARY_PATH like this:
$ export LD_LIBRARY_PATH=$HOME/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/mvapich2-gdr-2.3.7-44aoflbqebw3xbgp4rbncvrjwbjcwhs4/lib/:$LD_LIBRARY_PATH
Run OMB with this command:
$ ./bin/mpirun_rsh -np 2 -hostfile ~/hostfile ./libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency
Requesting generation of buildcache entries
- Operating System (RHEL7, CentOS7)
- Compiler (Version of gcc)
- Launcher (slurm, mpirun)
- OFED Version
- Cuda/ROCm Version (for GDR only)
- with or without mcast and/or openacc (for GDR only)
- Basic, Basic-XPMEM, Advanced, or Advanced-XPMEM (for X only)