Performance numbers of MVAPICH2 on Intel Haswell Architecture with Mellanox ConnectX-4 (07/23/18)

One Way Latency Unidirectional Bandwidth Bidirectional Bandwidth Notes
1.05 us 12380.52 MBps* 24628.38 MBps* The processes are bound to core 1 on both nodes
*MBps = Million Bytes per second

Machine Specifications

CPU Model CPU Core Info Memory Network Adapter Switch OS Network Stack
Intel E5-2687W 2x10 @ 3.1GHz 64GB Mellanox ConnectX-4 (100Gbps) Mellanox EDR Switch CentOS 7.4.1708 Mellanox OFED 4.2-1.2.0.0
osu_latency performance
osu_bw performance
osu_bibw performance
osu_put_latency performance
osu_put_bw performance
osu_put_bibw performance
osu_get_latency performance
osu_acc_latency performance

Performance numbers of MVAPICH2 on Intel Haswell Architecture with Mellanox ConnectX-4 (RoCE) (07/23/18)

One Way Latency Unidirectional Bandwidth Bidirectional Bandwidth Notes
1.03 us 11332.69 MBps* 21118.42 MBps* The processes are bound to core 1 on both nodes
*MBps = Million Bytes per second

Machine Specifications

CPU Model CPU Core Info Memory Network Adapter Switch OS Network Stack
Intel E5-2687W 2x10 @ 3.1GHz 64GB Mellanox ConnectX-4 (RoCE) (100Gbps) Back to Back CentOS 7.4.1708 Mellanox OFED 4.2-1.2.0.0
mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_put_bw
mv2 osu_put_bibw
mv2 osu_get_latency
mv2 osu_acc_latency

Performance numbers of MVAPICH2 on Intel Skylake Architecture with Intel Omni-Path (07/23/18)

One Way Latency Unidirectional Bandwidth Bidirectional Bandwidth Notes
0.94 us 12391.06 MBps* 24635.66 MBps* The processes are bound to core 1 on both nodes
*MBps = Million Bytes per second

Machine Specifications

CPU Model CPU Core Info Memory Network Adapter Switch OS Network Stack
Intel Xeon Platinum 8160 Processor 2x24 @ 2.1GHz 192GB Intel Omni-Path HFI (100Gbps) Intel Omni-Path Switch CentOS 7.4.1708 IFS 10.6
mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_get_latency
mv2 osu_acc_latency

Performance numbers of MVAPICH2 on Intel Haswell Architecture Intra-Node (07/23/18)

Communication MPI Latency Bandwidth Bidirectional Bandwidth Put Latency Get Latency Accumulate Latency Notes
Intra-Socket 0.22 us 15141.78 MBps* 26922.73 MBps* 0.05 us 0.05 us 0.08 us MV2_CPU_MAPPING=1:2
*MBps = Million Bytes per second
Inter-Socket 0.41 us 14432.20 MBps* 26033.54 MBps* 0.05 us 0.05 us 0.08 us MV2_CPU_MAPPING=1:11
*MBps = Million Bytes per second

Machine Specifications

CPU Model CPU Core Info Memory OS
Intel E5-2687W 2x10 @ 3.1GHz 64GB CentOS 7.4.1708

Intra-Socket

mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_get_latency
mv2 osu_acc_latency

Inter-Socket

mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_get_latency
mv2 osu_acc_latency

Performance numbers of MVAPICH2 on Intel KNL Architecture with Mellanox EDR (07/23/18)

One Way Latency Unidirectional Bandwidth Bidirectional Bandwidth Notes
2.22 us 12321.75 MBps* 24419.24 MBps* The processes are bound to core 1 on both nodes
*MBps = Million Bytes per second

Machine Specifications

CPU Model CPU Core Info Memory Network Adapter Switch OS Network Stack
Intel Xeon Phi 7250 1x68 @ 1.4GHz 96GB Mellanox EDR (100Gbps) Mellanox EDR Switch CentOS 7.4.1708 Mellanox OFED 4.2-1.2.0.0
mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_put_bw
mv2 osu_put_bibw
mv2 osu_get_latency
mv2 osu_acc_latency

Performance numbers of MVAPICH2 on Intel KNL Architecture with Intel Omni-Path (07/23/18)

One Way Latency Unidirectional Bandwidth Bidirectional Bandwidth Notes
2.33 us 10721.49 MBps* 14213.64 MBps* The processes are bound to core 7 on both nodes
*MBps = Million Bytes per second

Machine Specifications

CPU Model CPU Core Info Memory Network Adapter Switch OS Network Stack
Intel Xeon Phi 7250 1x68 @ 1.4GHz 96GB Intel Omni-Path HFI (100Gbps) Intel Omni-Path Switch CentOS 7.4.1708 IFS 10.6
mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_get_latency
mv2 osu_acc_latency

Performance numbers of MVAPICH2 on CascadeLake Architecture with HDR100 (06/01/20)

One Way Latency Unidirectional Bandwidth Bidirectional Bandwidth Notes
1.15 us 12307.07 MBps* 24525.78 MBps* The processes are bound to core 1 on both nodes
*MBps = Million Bytes per second

Machine Specifications

CPU Model CPU Core Info Memory Network Adapter Switch OS Network Stack
Intel(R) Xeon(R) Platinum 8280 2x28 @ 2.70GHz 192GB Mellanox HDR100 Mellanox FDR Switch CentOS 7.6.1810 Mellanox OFED 4.6-1.0.1
mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_put_bw
mv2 osu_put_bibw
mv2 osu_get_latency
mv2 osu_acc_latency

Performance numbers of MVAPICH2 on CascadeLake Architecture Intra-Node (06/01/20)

Communication MPI Latency Bandwidth Bidirectional Bandwidth Put Latency Get Latency Accumulate Latency Notes
Intra-Socket 0.21 us 11533.43 MBps* 23381.14 MBps* 0.05 us 0.05 us 0.07 us MV2_CPU_MAPPING=1:2
*MBps = Million Bytes per second
Inter-Socket 0.51 us 12467.66 MBps* 24984.85 MBps* 0.05 us 0.05 us 0.07 us MV2_CPU_MAPPING=1:11
*MBps = Million Bytes per second

Machine Specifications

CPU Model CPU Core Info Memory OS
Intel(R) Xeon(R) Platinum 8280 2x28 @ 2.70GHz 192GB CentOS 7.6.1810

Intra-Socket

mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_get_latency
mv2 osu_acc_latency

Inter-Socket

mv2 osu_latency
mv2 osu_bw
mv2 osu_bibw
mv2 osu_put_latency
mv2 osu_get_latency
mv2 osu_acc_latency