MVAPICH2-Virt 2.2

MVAPICH2-Virt provides a variety of MPI compilers to support applications written in different programming languages. Please use mpicc, mpif77, mpiCC, or mpif90 to compile applications. The correct compiler should be selected depending upon the programming language of your MPI application. These compilers are available in the MVAPICH2-Virt $HOME/bin directory. The MVAPICH2-Virt installation directory can also be specified by modifying the $PREFIX, then all the above compilers will also be present in the $PREFIX/bin directory.

For Singularity, please refer to The MVAPICH2 Userguide for the details about compiling applications with Singularity.

This section provides instructions on how to run applications with MVAPICH2-Virt. And this section shows the examples of running applications with mpirun_rsh. Please see the default MVAPICH2 userguide for more information on running with other process managers.

For Docker containers:

This command launches cpi on containers cont0 and cont1, two processes per container. By default ssh is used.

This command launches cpi on containers cont0 and cont1, two processes per container using rsh instead of ssh. More other options of mpirun_rsh can be obtained using the --help option.

For Singularity:

This command launches two Singularity containers on pyhsical node1 and node2 with optional runtime parameters $ENV, and run the executable cpi, which is inside the image Singularity.img.

For VMs: The corresponding commands are as follows:

This command launches cpi on VMs vm0 and vm1, two processes per VM. By default ssh is used.

This command launches cpi on VMs vm0 and vm1, two processes per VM using rsh instead of ssh. More information about how to use mpirun_rsh can be found in the default MVAPICH2 userguide.

As we just introduced, MVAPICH2-Virt enables IPC-SHM optimizations by default for containers. To use IVSHMEM based optimizations for VMs, the uses need to explicitely set MV2_VIRT_USE_IVSHMEM=1.

Setting MV2_CONT_USE_IPC=0 MV2_VIRT_USE_IVSHMEM=0 at the same time will disable both IPC-SHM and IVSHMEM based optimizations. By enabling either IPC-SHM or IVSHMEM optimization, users should be able to get better performance. Therefore, we strongly recommend users to enable this feature.

MVAPICH2-Virt can automatically select an IVSHMEM device if multiple devices are available in the virtual machines. MVAPICH2-Virt also provides the support for users to manually select an IVSHMEM device, by setting the parameter MV2_VIRT_IVSHMEM_DEVICE. For example, a user can use the following way to identify which device to be used in MPI jobs.

For OpenStack based Cloud environments, we provide scripts for integration with OpenStack by adminstrators to set up SR-IOV and IVSHMEM devices in VMs, which are needed by the MVAPICH2-Virt library. These scripts are compute-tools.sh, controller-tools.sh and ivshmem-tools.sh. Look for them under

Before running the above scripts, please run "source openrc.sh" to enable tenant/user credentials so that you have the permissions to execute different OpenStack commands. The detail steps are as follows:

This script should be executed on all compute nodes. It sets "pci_passthrough_whitelist" in OpenStack NOVA configuration with Vendor_ID and Product_ID for Virtual Function of Mellanox SR-IOV capable network device. After setting, it will restart the libvirtd and OpenStack NOVA compute services.

To manually set these two IDs (Vendor_ID and Product_ID) as arguments, adminstrators can execute the following command on compute nodes to get them first. They are most probably in the last square brackets. In our example, Vendor_ID is 15b3, and Product_ID is 1004.

The two input arguments are optional, which means the script can automatically detect these two IDs when executing on compute nodes. Please note that these two IDs are also needed when setting on controller node. The exampples of running this script is as below:

or

This script should be executed on the controller node only. It sets "pci_alias" for Virtual Function of Mellanox SR-IOV capable network device with Vendor_ID and Product_ID, which are the same as we get on compute nodes. Then it restarts related OpenStack services. After that, it defines the flavor for virtual machines, which includes flavor name, id, memory size, disk size, number of vcpus and number of virtual functions. As the following example shown, the first seven arguments are required and the last one is optional. Users can set the number of virtual functions they want to passthrough to each virtual machine by flag "-sriov". Its default value is 1.

The adminstrators can use this script to hotplug/unplug the IVSHMEM device to/from specified virtual machines. As the following example shown, it can accept four arguments in order to hotplug an IVSHMEM device to a virtual machine, which are "attach", virtual machine hostname, unique ivshmem tag, size of ivshmem (unit: MB). Note that the ivshmem tag for different users must be different. Otherwise, co-located virtual machines that different users launched will share the same IVSHMEM region with each other, which may results in unexpected data overwritten. And currently the value of ivshmem size will be capped by 256MB, due to the co-existence issue of SR-IOV and IVSHMEM devices inside one VM as we mentioned above. To unplug the IVSHMEM device, arguments "detach" and virtual machine hostname are needed.

The OSU Benchmarks can also be downloaded as a separate package from here. You can build the benchmarks using the following steps if mpicc is in your PATH. For example:

If mpicc is not in your path or you would like to use another particular version you can explicitly tell configure by setting CC. For example:

The OSU Benchmarks are run in the same manner as other MPI Applications. The following examples will use mpirun_rsh as the process manager. Please see the default MVAPICH2 userguide for more information on running with other process managers.

For Docker containers: The following example will measure the latency and bandwidth of communication between two Docker containers (node1-cont1 and node1-cont2) inside a same host.

The following example will measure the latency and bandwidth of communication inside one Docker container (node1-cont1 and node1-cont1).

The following example will measure the latency and bandwidth of communication between two Docker containers (node1-cont1 and node2-cont1) on two hosts.

For Singularity, The following example will measure the latency and bandwidth of communication between two Singularity containers on two different pyhsical nodes, node1 and node2.

For VMs: Similarly, the command will be as follows:

To run multiple jobs simultaneously on the same set of virtual machines, the administrator has to configure multiple IVSHMEM devices for each virtual machine. When the user wants to run multiple MPI jobs simultaneously, the user needs to explicitly specify one IVSHMEM device for each job by setting runtime parameter MV2_VIRT_IVSHMEM_DEVICE as mentioned in Support for User-specified IVSHMEM Device.