1. Launch Amazon Linux 2 Instance

Launch a c5n.18xlarge instance with Amazon Linux 2 AMI.

Follow the step 1-3 in this webpage:

2. Download RDMA Libraries

Download and install rdma-core libraries for mvapich2-x:

Download rdma-core from git-hub page
$ git clone https://github.com/amzn/rdma-core.git
install required packages
$ yum install autoconf automake libtool
$ yum install cmake gcc libnl3-devel libudev-devel make pkgconfig valgrind-devel
$ yum install epel-release
$ yum install cmake3 ninja-build pandoc
build with this command
$ cd rdma-core
$ bash build.sh
prepend library path to LD_LIBRARY_PATH like following
$ export LD_LIBRARY_PATH=/home/ec2-user/rdma-core/build/lib:$LD_LIBRARY_PATH

3. Install XPMEM

To run mvapich2-x, you need to install and load XPMEM as well.

Download the XPMEM module from the following Gitlab link

$ git clone https://gitlab.com/hjelmn/xpmem.git
Download and build xpmem
$ cd xpmem
$ ./autogen.sh
$ ./configure --prefix=/opt/xpmem --with-default-prefix=/opt/xpmem --with-module=/opt/xpmem/share/modules/xpmem
$ sudo make -j8 install
Load xpmem
$ sudo insmod /opt/xpmem/lib/modules/4.14.123-111.109.amzn2.x86_64/xpmem.ko
$ sudo chmod 666 /dev/xpmem
you can check if xpmem is loaded by following command and output:
$ lsmod | grep xpmem
xpmem                  32569  0
Note that in some cases you may need to prepend xpmem lib files to LD_LIBRARY_PATH:
$ export LD_LIBRARY_PATH=/opt/xpmem/lib:$LD_LIBRARY_PATH

4. Install MVAPICH2-X

Install mvapich2-x from rpm: (make sure you have sudo access)

Download mvapich2-x rpm with following command:
$ wget http://mvapich.cse.ohio-state.edu/download/mvapich/mv2x/mvapich2-x-intermediate-aws-ofed-gnu7.3.1-2.3-2.amzn2.x86_64.rpm
you can install mvapich2-x to default path /opt/mvapich2-x
$ rpm -Uvh --nodeps mvapich2-x-intermediate-aws-ofed-gnu7.3.1-2.3-2.amzn2.x86_64.rpm
or you can Install library using a prefix to specify install path.
$ rpm --prefix=/custom/install/prefix -Uvh --nodeps mvapich2-x-intermediate-aws-ofed-gnu7.3.1-2.3-2.amzn2.x86_64.rpm
If you do not have root permission or are on a system that does not use RPMs you can use rpm2cpio to extract the library.
$ rpm2cpio mvapich2-x-intermediate-aws-ofed-gnu7.3.1-2.3-2.amzn2.x86_64.rpm | cpio -id

5. Create More Instances

Now you can install HPC applications. To create more instances, make AMI from our created instance, launch new instances with the AMI so that you don’t need to re-install everything. Note that you need to repeat above step to load xpmem everytime when you launch a new instance or reboot an existed instance.

6. Example: How to Run OSU Micro-Benchmarks

OMB is installed as default in mvapich2-x install path, you can find OMB in ./libexec directory

go to mvapich2-x install path such as /opt/mvapich2-x/gnu7.3.1/aws-ofed/intermediate/mpirun

you may need to prepend mvapich2-x library to LD_LIBRARY_PATH like this:

$ export LD_LIBRARY_PATH=/opt/mvapich2-x/gnu7.3.1/aws-ofed/intermediate/mpirun/lib64/:$LD_LIBRARY_PATH

run OMB with this command:

$ ./bin/mpirun_rsh -np 2 -hostfile ~/hostfile ./libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency

7. More Information

For further details on various runtime parameters and optimizations, please refer to MVAPICH2-X userguide: