In scenarios where two IPoIB interfaces on the same host are configured on the same subnet, you must be sure to define the Linux routing tables for the desired behavior. This is because Linux may always use ib0 to send return packets even if the communication request is initiated over ib1. This also should be configured for proper Virtual Router Redundancy Protocol (VRRP) operation.

HostA:
ib0: 192.168.1.1
ib1: 192.168.2.1

HostB:
ib0: 192.168.1.2
ib1: 192.168.2.2

If a netmask, such as 255.255.0.0, is defined for the above hosts, communication over ib0 between the hosts may not work correctly. In this scenario, define the following routes on both hosts:

ip route add 192.168.1.0/24 via ib0
ip route add 192.168.2.0/24 via ib1

Alternatively, you can add these definitions to route-ib0/ib1 files.