The CN5000 SuperNIC ASIC has two ports but only the second port is active when a standard QSFP cable is used. This means tools like ibstat will show only the second port as Active, and the first port is Down. This behavior is expected. The OPX Provider will automatically select the second port when running MPI jobs. However, care must be taken when running IPoIB or verbs applications that the second port is selected. For example, running perftest ib_write_bw:

Server command:

ib_write_bw -F -a -d hfi1_0 -i 2 -q 4

Client command:

ib_write_bw -F -a -d hfi1_0 -i 2 -q 4 <server_ip_address>

Note in the above, -i 2 indicates to use the second port on the SuperNIC and -q 4 creates four queue pairs for improved scaling.

For applications that support CM queue pairs, the connection needs to be established over the IPoIB interface address. For example, add the -R option for perftest:

Server command

ib_write_bw -F -a -R -d hfi1_0 -i 2 -q 4

Client command

ib_write_bw -F -a -R -d hfi1_0 -i 2 -q 4 <server_IPoIB_address>