MVAPICH2-MIC-2.0 ---------------- MVAPICH2-MIC 2.0 binary release is based on MVAPICH2 2.0.1 and incorporates designs to optimize the communication for MIC(Phi)-based systems. MVAPICH2-MIC 2.0 supports the 3 modes of MIC usage: Offload, Native and Symmetric. It takes advantage of SCIF channel as well as a proxy based designs to boost the performance of Intra-MIC, Intra-node Host-MIC and Inter-nodes communications. Note that this release is for MIC-based systems, if your cluster does not have MICs, please use the default MVAPICH2 library. Even if your system has MICs and you want to run MPI or MPI+OpenMP programs using host processors only, please use the default MVAPICH2 library because this supports additional InfiniBand features which are not supported by MICs. For more details please refer to: http://mvapich.cse.ohio-state.edu/ Below are some instructions on how to use your MVAPICH2-MIC installation to run MIC enabled MPI jobs. Building Host and MIC Applications: -------------------------------------------------------------------------------- 1. Build Host application $ /opt/apps/intel14/mvapich2-mic/2.0/intel64/bin/mpicc -o foo foo.c 2. Build MIC application $ /opt/apps/intel14/mvapich2-mic/2.0/k1om/bin/mpicc -o foo-mic foo.c or alternatively the -mmic option can be used with the same mpicc (host version) $ /opt/apps/intel14/mvapich2-mic/2.0/intel64/bin/mpicc -mmic -o foo-mic foo.c Steps to Run an Application: -------------------------------------------------------------------------------- 1. Set required environment variables - MV2_IBA_HCA=mlx4_0 - (optional) MV2_USE_SCIF * toggle SCIF based optimizations (enabled by default) * please note that you must also disable the proxy (see section 6) when disabling SCIF - (optional) MV2_MIC_MAPPING * binding is disabled on the MIC by default but can be enabled by setting MV2_MIC_MAPPING in the same fashion as MV2_CPU_MAPPING * http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.0-userguide.html#x1-16700011.13 - (optional) MV2_MIC_INSTALL_PATH=/path/to/mvapich2-mic/install * we attempt to automatically detect path to the MIC build of MVAPICH2 library * if path to MIC build is not detected, we request the user to provide it using this variable * if user specifies the path using this variable; it overrides any automatically detected path It is suggested that a config file is created either system wide or per user to set these variables which will simplify launching for users. Example: $ cat ~/.mvapich2.conf MV2_MIC_INSTALL_PATH=/opt/apps/intel14/mvapich2-mic/2.0/k1om MV2_IBA_HCA=mlx4_0 The above example shows that a user config file was created in the default location to set both of the variables. A user can set MV2_USER_CONFIG to refer to a config file in a non-standard location. A system config file can be placed at /etc/mvapich2.conf with these contents if you want to prevent users from needing to worry about this. -------------------------------------------------------------------------------- 2. Create MPMD config file for job We are using the Multiple Process Multiple Data model in mpirun_rsh to launch jobs across MIC and host with the corresponding executables. This config will allow for running the osu_latency benchmark using one process on the host machine and the second process on the MIC. $ cat config -n 1 : /opt/apps/intel14/mvapich2-mic/2.0/intel64/libexec/mvapich2/osu_latency -n 1 : /opt/apps/intel14/mvapich2-mic/2.0/k1om/libexec/mvapich2/osu_latency -------------------------------------------------------------------------------- 3. Create hostfile for job Make sure the hosts in your hostfile are in corresponding order to the entries in your mpmd config file. $ cat hosts localhost:1 mic0:1 -------------------------------------------------------------------------------- 4. Launch job with mpirun_rsh $ /opt/apps/intel14/mvapich2-mic/2.0/intel64/bin/mpirun_rsh -config config \ -hostfile hosts # OSU MPI Latency Test # Size Latency (us) 0 4.71 1 6.19 2 6.21 4 6.16 8 6.19 16 6.59 32 6.78 64 5.42 128 5.72 256 6.62 512 7.51 1024 8.16 2048 9.43 4096 12.83 8192 23.74 16384 28.15 32768 16.97 65536 22.02 131072 31.83 262144 50.82 524288 89.36 1048576 166.21 2097152 319.58 4194304 626.18 -------------------------------------------------------------------------------- 5. Setting prefix/suffix of mic binary name - MV2_MIC_BINARY_PREFIX Set this environment variable to add a prefix to a host executable name to get a corresponding Intel Xeon Phi coprocessor executable file name. Example: Set different locations as the value for the MV2_MIC_BINARY_PREFIX environment variable to distinguish mic and host executable files: (host)$ /bin/mpicc -o osu_latency osu_latency.c (host)$ /bin/mpicc -o MIC/osu_latency osu_latency.c (host)$ MV2_MIC_INSTALL_PATH= /bin/mpirun_rsh -np 4 -hostfile MV2_IBA_HCA=mlx4_0 MV2_MIC_BINARY_PREFIX="MIC/" ./osu_latency In the example, "./osu_latency" binary will be launched on host nodes and "./MIC/osu_latency" binary on Intel Xeon Phi coprocessor nodes - MV2_MIC_BINARY_SUFFIX Set this environment variable to add a suffix to a host executable name to get a corresponding Intel Xeon Phi coprocessor executable file name Example: Set different names as the value for the MV2_MIC_BINARY_SUFFIX environment variable to distinguish mic and host executable files: (host)$ /bin/mpicc -o osu_latency osu_latency.c (host)$ /bin/mpicc -o osu_latency-mic osu_latency.c (host)$ MV2_MIC_INSTALL_PATH= /bin/mpirun_rsh -np 4 -hostfile MV2_IBA_HCA=mlx4_0 MV2_MIC_BINARY_SUFFIX="-mic" ./osu_latency In the example, "./osu_latency" binary will be launched on host nodes and "./osu_latency-mic" binary on Intel Xeon Phi coprocessor nodes ------------------------------------------------------------------------------ 6. Proxy daemon support The proxy daemon support can be used to alleviate poor bandwidth between MIC to remote MIC (on a different node or on a different socket) and MIC to remote HOST (on a different node). The daemon process proxies RDMA writes from the MIC through the HOST to improve large message bandwidth performance. This support is enabled by default. Users can disable it (not recommended) with the mpirun_rsh flag "-disable-mic-proxy" Note that the proxy is by default bound to the last core in each node. In order to achieve the best performance, please avoid the use of the last core. To do so, bind the application processes explicitly using MV2_CPU_MAPPING Example: $ /opt/apps/intel14/mvapich2-mic/2.0/intel64/bin/mpirun_rsh -config config \ -hostfile hosts 6.1. Active Proxy support Default proxy support. Env : MV2_MIC_PROXY_INTER_NODE_MODE values : 1 or 2 (default : 1) This parameter chooses inter node mic-to-mic communication active proxy method. The default method is optimal if Infiniband and MIC are connected to same processor I/O hub. It is recommended to select option 2 if Infiniband HCA and MIC are connected to two different processor I/O hub. 6.2. Passive proxy support Env : MV2_MIC_PROXY_PASSIVE Value : 0 or 1 (default : 0) This parameter disables Active proxy and enables Passive proxy. The passive proxy is recommended if the host CPUs are fully-subscribed (for instance, only if 16 MPI processes are to be launched on an 16-cores node; not when 15 or fewer cores are occupied). - (Optional) MV2_MAX_PROXY_SCIF_ISSUES * Default value: 8 * Number of concurrent SCIF operations that can be issued * Tune this for your application with Passive proxy design - (Optional) MV2_PROXY_BUFFER_SIZE * Default value: 64KB * Size of the chucks for the pipeline in passive proxy design * Tune this for your application with Passive proxy design