Skip to main content

Cornelis Technical Documentation

6.8.7. Enabling 64KB MTU for Connected Mode IPoIB

For optimal performance in Connected Mode, enable a 64KB MTU.

  1. Modify the ifcfg-ib0 file:

    On RHEL:

    MTU=65520 
CONNECTED_MODE=yes

    On SLES:

    MTU=65520
IPOIB_MODE=connected

  2. Apply the changes:

    ip link set ib0 down 
modprobe -r ib_ipoib 
modprobe ib_ipoib 
ip link set ib0 up

  3. Check if Connected Mode is enabled:

    cat /sys/class/net/ib0/mode 
connected

  4. Check if MTU is set to 65520:

    cat /sys/class/net/ib0/mtu 
65520

    Example output from ifconfig:

    ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 65520
       inet 10.143.0.155 netmask 255.255.0.0 broadcast 10.143.255.255
       inet6 fe80::d206:6a02:100:c6 prefixlen 64 scopeid 0x20<link>

    The output confirms that:

    • The interface ib0 is in Connected Mode.

    • The MTU is correctly set to 65520 bytes.

These settings are critical for achieving maximum bandwidth performance with IPoIB using connected mode.