Skip to main content

Cornelis Technical Documentation

3.5.3. Configuring Boot Over Fabric

3.5.3.1. UEFI Secure Boot Support

OPX Software includes support for UEFI Secure Boot, both in the UEFI SuperNIC firmware environment and in Linux.

3.5.3.1.1. UEFI Firmware Support

All UEFI drivers shipped as part of the CN5000 OPX Software are signed to support UEFI Secure Boot using the Microsoft UEFI CA signing process.

3.5.3.1.2. Linux Support

Kernel modules shipped with major Linux distributions are signed using the keys for those distributions.

Kernel modules shipped as part of the OPX Software installation suite are signed using a Cornelis key. The Cornelis public key is automatically added to the Machine Owner Key (MOK) list in the UEFI firmware environment with no action required by the user.

The following MOK utility can be used to list the currently enrolled keys in the MOK list keyring:

mokutil --list-enrolled

For more information about UEFI Secure Boot support in RHEL, refer to the RHEL documentation at https://docs.redhat.com/en.

For more information about UEFI Secure Boot support in SLES, refer to the SLES documentation at https://documentation.suse.com/.

3.5.3.1.2.1. Limitations

When UEFI Secure Boot mode is enabled, the SuperNIC firmware cannot be updated or downgraded.

3.5.3.2. Configuring PXE Boot Over IPoIB

The following sections provide instructions for setting up PXE Boot on RHEL 9.x and RHEL 10.x operating systems.

Important

  • The standard installation initrd.img provided by Red Hat does not bundle all drivers required to establish IPoIB communications over the CN5000 SuperNIC. The RAM disk image must be modified before attempting to boot over fabric.

  • The target OS/version initrd.img must be compiled and modified on a machine running the matching OS and kernel version.

3.5.3.2.1. Overview and Architecture

Boot Over Fabric requires at least two nodes connected to an Omni-Path fabric (either directly back-to-back or via a CN5000 Switch). One node serves as the PXE Server, while the other is the Client Node where the operating system will be installed. Communication between both nodes occurs over IP over InfiniBand (IPoIB).

The PXE boot exchange proceeds through four distinct stages:

  1. DHCP Stage: The Client broadcasts a DHCP request across the IPoIB fabric partition. The DHCP server returns an IP address, the TFTP server address, and the bootloader file path (uefi/grubx64.efi).

    Note

    For secure PXE Boot, use shimx64.efi instead of grubx64.efi.

  2. TFTP Stage: The Client contacts the TFTP server and downloads the UEFI bootloader image (grubx64.efi) and boot configuration (grub.cfg).

  3. Kernel and initrd Stage: The Client executes GRUB, downloads the installer kernel (vmlinuz) and initial RAM disk (initrd_CN5K.img) from the TFTP server, and boots the kernel.

  4. OS Installation Stage (HTTP): The OS installer initializes on the Client and retrieves the complete OS installation tree, runtime image (install.img), and packages from the HTTP server via inst.repo.

3.5.3.2.2. Prerequisites and Fabric Setup

Verify fabric readiness and assign static addressing on the PXE Server before configuring daemon services.

Fabric Manager and Port Verification

Ensure the Fabric Manager is running on the fabric and the SuperNIC port is active by running opainfo:

opainfo | grep -iE 'PortState|LinkSpeed'

Expected output includes:

PortState: Active
LinkSpeed Act: 100Gb   #or target link rate
Configure Server IPoIB Address

The PXE environment operates on partition key (P_Key) 0x8001. The server must bind 192.168.100.1/24 to the child interface representing this partition (for example, ibs5d1 or ibp13s0d1):

  1. Locate the child interface corresponding to pkey 0x8001 with an active carrier:

    for d in /sys/class/net/ib*; do 
  n=$(basename "$d") 
  echo "$n pkey=$(cat $d/pkey 2>/dev/null) carrier=$(cat $d/carrier 2>/dev/null)"; 
done

  2. Assign the static IP address and bring up the interface:

    ip addr add 192.168.100.1/24 dev <server-ib-child> 
ip link set <server-ib-child> up

    Note

    <server-ib-child> represents the server's local IPoIB partition child interface, for example, ibs5d1.

  3. Confirm datagram mode and address assignment:

    ip -4 addr show dev <server-ib-child> | grep inet 
cat /sys/class/net/<server-ib-child>/mode

Expected mode: datagram

3.5.3.2.3. Configure HTTP Installation Server

The HTTP server hosts the extracted OS repository and stage2 installation runtime images.

  1. Install Apache HTTP Server:

    dnf install -y httpd 
mkdir -p /var/www/html/netinstall

  2. Create configuration /etc/httpd/conf.d/netinstall.conf:

    Alias /netinstall /var/www/html/netinstall 
<Directory /var/www/html/netinstall> 
  Options Indexes FollowSymLinks 
  Require all granted 
</Directory>

  3. Bind Apache exclusively to the IPoIB interface. In /etc/httpd/conf/httpd.conf, comment out Listen 80 and add:

    # Listen 80
Listen 192.168.100.1:80

  4. Enable and start Apache:

    systemctl enable --now httpd.service
3.5.3.2.4. Set Up Procedure for RHEL 9.x PXE Server

This section details how to configure the PXE server, inject CN5000 SuperNIC drivers into the initial RAM disk, and stage boot assets using the Internet Systems Consortium (ISC) DHCP server and standalone TFTP daemon for RHEL 9.x clients.

Configure the DHCP Server

  1. Install the ISC DHCP server:

    dnf install -y dhcp-server

  2. Edit /etc/dhcp/dhcpd.conf:

    # DHCP Server Configuration file.
#
option space pxelinux;
option pxelinux.magic code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;
option architecture-type code 93 = unsigned integer 16;
allow booting;
allow bootp;

subnet 192.168.100.0 netmask 255.255.255.0 {
  range 192.168.100.2 192.168.100.230;
  option subnet-mask 255.255.255.0;
  option broadcast-address 192.168.100.255;
  option routers 192.168.100.1;

  class "pxeclients" {
    match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
    next-server 192.168.100.1;
    if option architecture-type = 00:07 {
         filename "uefi/grubx64.efi";
    } else {
         filename "pxelinux/pxelinux.0";
    }
  }
}

  3. Enable and start the DHCP service:

    systemctl enable --now dhcpd.service
Configure the TFTP Server

  1. Install TFTP server packages:

    dnf install -y tftp tftp-server

  2. Create systemd socket unit /etc/systemd/system/tftp.socket:

    [Unit] 
Description=TFTP Trivial File Transfer Protocol Socket 

[Socket] 
ListenDatagram=69 

[Install] 
WantedBy=sockets.target

  3. Create systemd service unit /etc/systemd/system/tftp.service:

    [Unit]
Description=Tftp Server
Requires=tftp.socket

[Service]
# The user the server will run as
User=root

# The TFTP server executable and its arguments
ExecStart=/usr/sbin/in.tftpd -c -p -s /var/lib/tftpboot
StandardInput=socket
StandardError=journal

[Install]
WantedBy=multi-user.target
Also=tftp.socket

  4. Reload systemd and start the TFTP socket:

    systemctl daemon-reload 
systemctl enable --now tftp.socket tftp.service
Prepare the Client initrd Image

  1. Mount the OS ISO and unpack the initrd.

    1. Mount the installation ISO:

      mount -o loop RHEL-9.x-x86_64-dvd.iso /media/

    2. Create the working directory and switch to it:

      mkdir -p /tmp/images/initrd  
cd /tmp/images/initrd

    3. Copy the initrd image for backup/local reference:

      cp /media/images/pxeboot/initrd.img .

    4. Decompress and unpack the initrd filesystem:

      xz -dc initrd.img | cpio -idmv

  2. Inject Drivers.

    1. Define the Environment Variables:

      KERNEL_VERSION=5.14.0_687.5.3.el9_8.x86   # match the target OS ISO 
INITRD=/tmp/images/initrd
OPXS=/lib/modules/$KERNEL_VERSION/extra/opxs-kernel-updates
BASE=$INITRD/lib/modules/$KERNEL_VERSION/kernel/drivers/infiniband

    2. Create required driver directory structure:

      mkdir -p $BASE/hw/hfi1 $BASE/sw/rdmavt $BASE/core $BASE/ulp/ipoib

    3. Remove inbox compressed drivers, if present, to prevent precedence conflicts:

      rm -f $BASE/hw/hfi1/hfi1.ko* $BASE/sw/rdmavt/rdmavt.ko* \
      $BASE/core/ib_uverbs.ko* $BASE/ulp/ipoib/ib_ipoib.ko*

    4. Copy the SuperNIC drivers UNCOMPRESSED

      Note

      This avoids CRC64 decompression failure status 6 caused by default xz.

      cp $OPXS/hfi1.ko   $BASE/hw/hfi1/
cp $OPXS/rdmavt.ko $BASE/sw/rdmavt/

    5. Decompress and inject required core InfiniBand and IPoIB dependencies to plain .ko:

      xz -dc /lib/modules/$KERNEL_VERSION/kernel/drivers/infiniband/core/ib_uverbs.ko.xz \
    > $BASE/core/ib_uverbs.ko
xz -dc /lib/modules/$KERNEL_VERSION/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko.xz \
    > $BASE/ulp/ipoib/ib_ipoib.ko

    6. Copy SuperNIC link training firmware files:

      mkdir -p $INITRD/lib/firmware/updates
for fw in hfi1_dc8051.fw hfi1_fabric.fw hfi1_pcie.fw hfi1_sbus.fw; do
    cp /lib/firmware/updates/$fw $INITRD/lib/firmware/updates/
done

    7. Regenerate module dependencies inside the unpacked initrd:

      depmod -b $INITRD $KERNEL_VERSION

    8. Verify injected modules are uncompressed and present:

      for f in hw/hfi1/hfi1 sw/rdmavt/rdmavt core/ib_uverbs ulp/ipoib/ib_ipoib; do
    test -f $BASE/$mod.ko && echo "$f.ko OK" || echo "MISSING $f.ko"
    test -e $BASE/$f.ko.xz && echo "LEFTOVER $f.ko.xz (BAD)"
done

    9. Verify modules.dep was successfully generated with hfi1:

      grep -q "/hfi1\.ko" $INITRD/lib/modules/$KERNEL_VERSION/modules.dep && echo 
"hfi1 in modules.dep"

  3. Ensure ib_ipoib loads at boot:

    mkdir -p $INITRD/lib/modules-load.d 
printf "# Load ib_ipoib.ko at boot\nib_ipoib\n"  \
   > $INITRD/lib/modules-load.d/ib_ipoib.conf

    Note

    • Write this file with truncation >, not append >>, to avoid duplicate entries when the procedure is rerun.

    • No udev rule is needed to create the partition child interface. Once ib_ipoib loads and the SM programs the port's P_Key table, the IPoIB stack automatically creates the matching child interface (e.g., ibs8d1 for P_Key 0x8001). The grub ip= argument in the next section points DHCP at that auto-created interface.

  4. Repack the RAM disk:

    cd $INITRD
find . | cpio -o -c -R root:root | gzip -9 > ../initrd_CN5K.img

mkdir -p /var/lib/tftpboot/uefi
cp ../initrd_CN5K.img /var/lib/tftpboot/uefi/
cp /media/images/pxeboot/vmlinuz /var/lib/tftpboot/uefi/
Stage Install Tree and GRUB Configuration

  1. Publish the ISO contents to HTTP.

    1. Define target OS minor release directory (for example, RHEL9.5, RHEL9.8):

      OS_DIR="RHEL9.5"

    2. Create netinstall directory for the target release:

      mkdir -p /var/www/html/netinstall/ISO/$OS_DIR

    3. Copy installation files from the mounted ISO:

      rsync -av /media/ /var/www/html/netinstall/ISO/$OS_DIR

    4. Unmount the temporary loopback mount:

      umount /media

    5. Set ownership and permissions for Apache:

      chmod -R 755 /var/www/html/netinstall/ISO/$OS_DIR
chown -R apache:apache /var/www/html/netinstall/ISO/$OS_DIR

    6. Stage the UEFI bootloader binary in TFTP boot directory:

      cp /var/www/html/netinstall/ISO/$OS_DIR/EFI/BOOT/grubx64.efi /var/lib/tftpboot/uefi/

  2. Configure /var/lib/tftpboot/uefi/grub.cfg.

    • RHEL 9.5 Entry:

      set timeout=20
menuentry 'RHEL 9.5 Install (SuperNIC IPoIB)' {
    linuxefi uefi/vmlinuz net.ifnames=0 biosdevname=0 rd.neednet=1 rd.net.timeout.carrier=60 rd.driver.pre=hfi1,ib_ipoib ip=ib1:dhcp ip.bootdev=ib1 bootproto=dhcp inst.repo=http://192.168.100.1/netinstall/ISO/RHEL9 iommu=pt
    initrdefi uefi/initrd_CN5K.img
}

    • RHEL 9.8 Entry:

      Replace <IFACE> with the client's PCI altname, for example, ibp13s0d1.

      set timeout=20
menuentry 'RHEL 9.8 Install (SuperNIC IPoIB)' {
    linuxefi uefi/vmlinuz biosdevname=0 rd.neednet=1 rd.net.timeout.carrier=60 rd.driver.pre=hfi1,ib_ipoib ip=<IFACE>:dhcp bootproto=dhcp inst.waitfornet=60 inst.repo=http://192.168.100.1/netinstall/ISO/RHEL9 iommu=pt
    initrdefi uefi/initrd_CN5K.img
}

      Note

      <IFACE> represents the client's IPoIB partition child interface as discovered inside the installer initrd, for example, ib1 or ibp13s0d1.

      To determine the <IFACE> on the PXE partition, list the client's IB netdevs in ifindex order and pick the link-up P_Key 0x8001 child (non-zero dev_port); its ib<N> name is its position in that ordering:

      for d in /sys/class/net/ib*; do
  n=$(basename "$d")
  printf "ifindex=%s name=%s pkey=%s carrier=%s dev_port=%s\n" \
    "$(cat $d/ifindex)" "$n" "$(cat $d/pkey)" "$(cat $d/carrier)" "$(cat $d/dev_port)"
done | sort

      The link-up (carrier=1) pkey=0x8001 child (dev_port!=0) is ib<N> at its ifindex position.

Verify the Server Before Booting

  1. Confirm the DHCP/TFTP service and the HTTP install tree from the PXE server:

    • dhcpd.service is active and bound to the ib interface:

      systemctl is-active dhcpd.service
journalctl -u dhcpd.service --no-pager -n 10

    • tftp.socket is active and socket listener is bound to port 69:

      systemctl is-active tftp.socket
ss -ulnp | grep :69

    • Boot Files are in the TFTP Staging Directory:

      ls -lh /var/lib/tftpboot/uefi/

  2. Test HTTP Access and Installation Image:

    • HTTP install tree (expect HTTP/1.1 200 OK):

      curl -I http://192.168.100.1/netinstall/ISO/$OS_DIR/
curl -I http://192.168.100.1/netinstall/ISO/$OS_DIR/.treeinfo
curl -I http://192.168.100.1/netinstall/ISO/$OS_DIR/images/install.img

    • install.img size (~600–1200 MB depending on release):

      curl -sI http://192.168.100.1/netinstall/ISO/$OS_DIR/images/install.img | grep -i content-length
3.5.3.2.5. Set Up Procedure for RHEL 10.x PXE Server

This section details the PXE server configuration and boot image staging for RHEL 10.x clients, using dnsmasq for integrated DHCP and TFTP services and injecting uncompressed SuperNIC drivers to comply with Linux Kernel module integrity checks.

Configure Combined DHCP and TFTP Server (dnsmasq)

Note

Kea DHCP Incompatibility: Kea cannot be used for IPoIB PXE boot. Its raw socket rejects non-802.3 InfiniBand framing, and its UDP mode fails to capture RFC 4390 broadcast traffic. dnsmasq handles IPoIB natively while providing integrated TFTP.

  1. Install dnsmasq:

    dnf install -y dnsmasq 
mkdir -p /etc/dnsmasq.d /var/lib/tftpboot/uefi

  2. Create /etc/dnsmasq.d/ipoib-pxe.conf (substituting <server-ib-child> with your partition interface, e.g., ibs5d1).

    1. Disable DNS listener (port 0); serve only DHCP and TFTP:

      port=0
interface=<server-ib-child>
bind-interfaces
dhcp-authoritative
log-dhcp

    2. Dynamic IPoIB range and gateway:

      dhcp-range=192.168.100.2,192.168.100.230,255.255.255.0,1h
dhcp-option=option:router,192.168.100.1

    3. UEFI x86_64 architecture matching (RFC 4578 codes 7 and 9) :

      dhcp-match=set:efi-x86_64,option:client-arch,7
dhcp-match=set:efi-x86_64,option:client-arch,9
dhcp-boot=tag:efi-x86_64,uefi/grubx64.efi,,192.168.100.1
dhcp-boot=tag:!efi-x86_64,pxelinux/pxelinux.0,,192.168.100.1

    4. Integrated TFTP configuration:

      enable-tftp
tftp-root=/var/lib/tftpboot

  3. Test syntax and start dnsmasq:

    dnsmasq --test 
systemctl enable --now dnsmasq 
journalctl -u dnsmasq --no-pager -n 5   # expect: sockets bound exclusively to interface ...

  4. (Optional) Pin a static IP using RFC 4361 Client-Identifier.

    Because IPoIB lacks Ethernet MAC addresses, chaddr is zeroed (htype=32, hlen=0) per RFC 4390. Pin reservations using the Option 61 identifier:

    tcpdump -i <server-ib-child> -vvv -s0 'port 67 or port 68'

    The Client-ID (61) begins with the IPoIB hardware type 0x20 (32) followed by the 8-byte port GUID tail, for example, hardware-type 32, d0:06:6a:02:01:00:3b:1a.

    Then add a reservation to /etc/dnsmasq.d/ipoib-pxe.conf and restart:

    dhcp-host=id:20:d0:06:6a:02:01:00:3b:1a,192.168.100.50,cn5k-client 
systemctl restart dnsmasq
Prepare the Client initrd Image

Important

Inject Modules Uncompressed: Linux Kernel 6.12 rejects .ko.xz compressed with default CRC64 (decompression failed with status 6 / Invalid argument). All injected modules must be installed as uncompressed .ko files.

  1. Mount the RHEL 10.x ISO and extract initrd.img.

    1. Download and mount the ISO (where x is the target RHEL version) :

      wget <RHEL-10.x-dvd.iso>
mount -t iso9660 -o loop RHEL-10.x-dvd.iso /media/

    2. Extract the installer initrd into a work tree:

      mkdir -p /tmp/images/initrd
cd /tmp/images/initrd
xz -dc /media/images/pxeboot/initrd.img | cpio -idmvAll paths below use $KERNEL_VERSION (the installer kernel for the target OS).

  2. Set environment paths and inject SuperNIC kernel modules uncompressed.

    1. Set the Environment Variables:

      KERNEL_VERSION=6.12.0-124.8.1.el10_1.x86_64   # Match your target ISO kernel
INITRD=/tmp/images/initrd
OPXS=/lib/modules/$KERNEL_VERSION/extra/opxs-kernel-updates
BASE= $INITRD/lib/modules/$KERNEL_VERSION/kernel/drivers/infiniband

    2. Create required driver directory structure:

      mkdir -p $BASE/hw/hfi1 $BASE/sw/rdmavt $BASE/core $BASE/ulp/ipoib

    3. Remove inbox compressed drivers, if present:

      rm -f $BASE/hw/hfi1/hfi1.ko.xz $BASE/sw/rdmavt/rdmavt.ko.xz

    4. Copy the SuperNIC drivers UNCOMPRESSED:

      cp $OPXS/hfi1.ko   $BASE/hw/hfi1/
cp $OPXS/rdmavt.ko $BASE/sw/rdmavt/

    5. Decompress required core dependencies to plain .ko:

      # ib_uverbs (rdmavt dependency)
xz -dc /lib/modules/$KERNEL_VERSION/kernel/drivers/infiniband/core/ib_uverbs.ko.xz > $BASE/core/ib_uverbs.ko
rm -f $BASE/core/ib_uverbs.ko.xz

# ib_ipoib (creates the IPoIB netdev)
xz -dc /lib/modules/$KERNEL_VERSION/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko.xz > $BASE/ulp/ipoib/ib_ipoib.ko
rm -f $BASE/ulp/ipoib/ib_ipoib.ko.xz

  3. Update module dependencies and stage the SuperNIC link firmware.

    1. Regenerate dependency map:

      depmod -b $INITRD $KERNEL_VERSION

    2. Copy link training firmware:

      mkdir -p $INITRD/lib/firmware/updates
for fw in hfi1_dc8051.fw hfi1_fabric.fw hfi1_pcie.fw hfi1_sbus.fw; do
    cp /lib/firmware/updates/$fw $INITRD/lib/firmware/updates/
done

    3. Force ib_ipoib loading at boot:

      mkdir -p $INITRD/lib/modules-load.d
printf "# Load ib_ipoib.ko at boot\nib_ipoib\n" > $INITRD/lib/modules-load.d/ib_ipoib.conf

      Note

      • Write this file with truncation >, not append >>, to avoid duplicate entries when the procedure is rerun.

      • No udev rule is needed to create the partition child interface. Once ib_ipoib loads and the SM programs the port's P_Key table, the IPoIB stack automatically creates the matching child interface (e.g., ibs8d1 for P_Key 0x8001). The grub ip= argument in the next section points DHCP at that auto-created interface.

  4. Verify injected modules before repacking.

    • hfi1/rdmavt are present as plain .ko, no leftover .ko.xz:

      for f in hw/hfi1/hfi1 sw/rdmavt/rdmavt core/ib_uverbs ulp/ipoib/ib_ipoib; do
    test -f $BASE/$f.ko && echo "$f.ko OK" || echo "MISSING $f.ko"
    test -e $BASE/$f.ko.xz && echo "ERROR: Leftover .ko.xz detected in $f"
done

    • hfi1 vermagic matches the installer kernel:

      modinfo -F vermagic $BASE/hw/hfi1/hfi1.ko # must start with $KERNEL_VERSION

    • depmod registered hfi1:

      grep -q "/hfi1\.ko" $INITRD/lib/modules/$KERNEL_VERSION/modules.dep && echo "modules.dep OK"

  5. Repack initrd and stage boot files:

    cd $INITRD
find . | cpio -o -c -R root:root | gzip -9 > ../initrd_CN5K.img

mkdir -p /var/lib/tftpboot/uefi
cp $INITRD/initrd_CN5K.img /var/lib/tftpboot/uefi/
cp /media/images/pxeboot/vmlinuz /var/lib/tftpboot/uefi/
Stage Install Tree and GRUB Configuration

  1. Copy ISO contents to HTTP document root.

    1. Define target OS minor release directory (for example, RHEL10.1):

      OS_DIR="RHEL10.1"

    2. Create netinstall directory for the target release:

      mkdir -p /var/www/html/netinstall/ISO/$OS_DIR

    3. Copy installation files from the mounted ISO:

      rsync -av /media/ /var/www/html/netinstall/ISO/$OS_DIR

    4. Unmount the temporary loopback mount:

      umount /media

    5. Set ownership and permissions for Apache:

      chmod -R 755 /var/www/html/netinstall/ISO/$OS_DIR
chown -R apache:apache /var/www/html/netinstall/ISO/$OS_DIR

    6. Stage the UEFI bootloader binary in TFTP boot directory:

      cp /var/www/html/netinstall/ISO/$OS_DIR/EFI/BOOT/grubx64.efi /var/lib/tftpboot/uefi/

    7. Stage the UEFI bootloader binary in TFTP boot directory:

      cp /var/www/html/netinstall/ISO/$OS_DIR/EFI/BOOT/grubx64.efi /var/lib/tftpboot/uefi/

  2. Configure /var/lib/tftpboot/uefi/grub.cfg:

    cat << 'EOF' > /var/lib/tftpboot/uefi/grub.cfg
set default="0"
set timeout=20

menuentry 'RHEL 10.x Install (SuperNIC IPoIB)' {
    linuxefi uefi/vmlinuz net.ifnames=0 biosdevname=0 rd.neednet=1 \ rd.net.timeout.carrier=120 rd.net.timeout.dhcp=60 rd.driver.pre=hfi1,ib_ipoib \ ip=ib1:dhcp ip.bootdev=ib1 bootproto=dhcp inst.waitfornet=180 \ inst.repo=http://192.168.100.1/netinstall/ISO/RHEL10.1 iommu=pt
    initrdefi uefi/initrd_CN5K.img
}
EOF

    For a single-SuperNIC node with one active port, the base port is ib0 and the P_Key 0x8001 child is ib1.

    Note

    <IFACE> represents the client's IPoIB partition child interface as discovered inside the installer initrd, for example, ib1 or ibp13s0d1.

    To determine the <IFACE> on the PXE partition, list the client's IB netdevs in ifindex order and pick the link-up P_Key 0x8001 child (non-zero dev_port); its ib<N> name is its position in that ordering:

    for d in /sys/class/net/ib*; do
  n=$(basename "$d")
  printf "ifindex=%s name=%s pkey=%s carrier=%s dev_port=%s\n" \
    "$(cat $d/ifindex)" "$n" "$(cat $d/pkey)" "$(cat $d/carrier)" "$(cat $d/dev_port)"
done | sort

    The link-up (carrier=1) pkey=0x8001 child (dev_port!=0) is ib<N> at its ifindex position.

  3. Reload HTTP:

    systemctl restart httpd.service
Verify the Server Before Booting

Confirm the DHCP/TFTP service and the HTTP install tree from the PXE server:

  • dnsmasq is active and bound to the ib interface

    systemctl is-active dnsmasq
journalctl -u dnsmasq --no-pager -n 20 | grep 'sockets bound exclusively to interface'

  • HTTP install tree (expect HTTP/1.1 200 OK)

    curl -I http://192.168.100.1/netinstall/ISO/$OS_DIR/.treeinfo
curl -I http://192.168.100.1/netinstall/ISO/$OS_DIR/images/install.img
curl -I http://192.168.100.1/netinstall/ISO/$OS_DIR/

  • install.img size (~600–1200 MB depending on release)

    curl -sI http://192.168.100.1/netinstall/ISO/$OS_DIR/images/install.img | grep -i content-length
3.5.3.2.6. Set Up Client Firmware and Boot Execution

  1. Access the Client BIOS/UEFI Setup utility.

  2. Enable the following settings:

    • UEFI Network Stack: Enabled

    • IPv4 PXE Support: Enabled

    • Cornelis SuperNIC Option ROM: Enabled

  3. Set UEFI PXE (IPv4) as the first boot device in the boot order.

  4. On the PXE Server, monitor service interaction in real time:

    • RHEL 9.x Server:

      journalctl -u dhcpd.service -f

    • RHEL 10.x Server:

      journalctl -u dnsmasq -f

  5. Monitor HTTP repository downloads:

    tail -f /var/log/httpd/access_log
3.5.3.2.7. Post-Install OPX Software and IPoIB Interface Setup

During the PXE boot phase, network configuration lives in temporary memory (initrd). Once booted into the target OS, the client must be configured to persistently obtain or assign its IP address on the SuperNIC interface.

In modern Linux distributions (RHEL 9.x and RHEL 10.x), persistent network configuration is managed via NetworkManager keyfiles located in /etc/NetworkManager/system-connections/.

  1. Configure persisent IpoIB NetworkManager Connection

    Because communication occurs over partition 0x8001, configure a child partition connection bound to the physical parent interface:

  2. Define interface names (for example, parent=ib0, child=ib0.8001)

    PARENT_IF="ib0"
CHILD_IF="ib0.8001"
PKEY="0x8001"
UUID=$(uuidgen)

  3. Create NetworkManager keyfile for the IPoIB child partition

    cat << EOF > /etc/NetworkManager/system-connections/$CHILD_IF.nmconnection
[connection]
id=$CHILD_IF
uuid=$UUID
type=infiniband
interface-name=$CHILD_IF

[infiniband]
transport-mode=datagram
parent=$PARENT_IF
p-key=$PKEY

[ipv4]
method=auto

[ipv6]
addr-gen-mode=eui64
method=auto
EOF

  4. Lock down file permissions to root (mandatory for NetworkManager keyfiles)

    chmod 600 /etc/NetworkManager/system-connections/$CHILD_IF.nmconnection

  5. Reload NetworkManager and bring up the partition interface

    nmcli connection reload
nmcli connection up $CHILD_IF

  6. Install OPX Software and Configure Partitions

    Because the clean OS installation from the ISO does not contain the updated Omni-Pathsoftware stack, the installed OS requires final driver and interface initialization:

    1. Install the OPX Software.

      Install the OPX Software package on the client so that native SuperNIC drivers (hfi1, rdmavt) and link firmware are loaded permanently across standard kernel updates. Refer to “Install the OPX Software”.

    2. Configure Fabric Subnets (Parent vs. Child Interfaces):

      • Parent interface (for example, ib0): Bound to the default fabric partition (0x7fff/0xffff).

      • Child interface (for example, ib0.8001 or ibs1d1): Bound to partition key 0x8001.

        Important

        Child interfaces must be assigned to a different IP subnet than their parent interfaces. Because they operate on distinct fabric partitions, assigning them to the same IP subnet will cause packet forwarding and routing failures due to P_Key mismatches.

3.5.3.2.8. Troubleshooting PXE Boot

Symptom/Error

Root Cause

Corrective Action

Dracut emergency shell; dmesg reports decompression failed with status 6

On RHEL 10.x, kernel module was injected using .ko.xz (CRC64).

Inject modules as plain uncompressed .ko files. Ensure no leftover .ko.xz files remain.

modprobe: FATAL: Module hfi1 not found

depmod was not executed after copying new modules into initrd.

Run depmod -b $INITRD $KERNEL_VERSION before repacking.

Client hangs at DHCP discovery; no packets on server

dnsmasq configured with dhcp-socket-type=udp, blocking broadcast packets.

Remove dhcp-socket-type=udp from ipoib-pxe.conf.

Client receives IP via Ethernet rather than SuperNIC

Bare ip=dhcp used in GRUB, causing an Ethernet race condition.

Explicitly define the boot interface in GRUB: ip=ib1:dhcp ip.bootdev=ib1

Installer fails stage2 image download

IPoIB fabric link training and broadcast group join lagged timeouts.

Increase GRUB boot parameters: rd.net.timeout.carrier=120 rd.net.timeout.dhcp=60 inst.waitfornet=180

SuperNIC link fails to achieve Active state

Firmware missing from initrd or Fabric Manager (opafm) not running.

Verify opainfo on fabric; verify hfi1_*.fw files exist in /lib/firmware/updates/