Skip to main content

Cornelis Technical Documentation

4.4.4. Configuring Dual Rails for Dual Subnets

A common multi-rail, multi-subnet scenario is to set up dual rails for dual subnets where you configure two SuperNICs in each host to two different subnets to provide failover and load-balancing functions.

4.4.4.1. Assumptions

  • SuperNICs have been installed in the host servers.

  • SuperNICs have been cabled as shown below:

    Figure 71. SuperNIC Cabling
    SuperNIC Cabling


  • Similarly, the SuperNICs in the Fabric Managers have been cabled as shown below:

    Figure 72. SuperNIC in the Fabric Manager Cabling
    SuperNIC in the Fabric Manager Cabling


4.4.4.2. Procedures

Perform the following steps to set up the primary Fabric Manager and standby Fabric Managers:

  1. Stop all standby Fabric Managers using systemctl stop opafm.

  2. Stop the primary Fabric Manager using systemctl stop opafm.

  3. Open the /etc/opa-fm/opafm.xml file on the primary Fabric Manager for editing.

  4. Search for "<Shared>" to review the fm0 settings.

    An example of the shared instance for fm0 with key items in bold is shown below.

    <!-- Shared Instance config, applies to all components: SM, PM and FE -->
<Shared>
  <!-- Fm.Shared.Start controls overall startup of the Instance. -->
  <!-- If 0, none of the components in the Instance are started. -->
  <!-- If 1, instance is enabled and Fm.Sm.Start, Fm.Pm.Start, etc -->
  <!-- control startup of each manager.  The default for each manager -->
  <!-- is defined by Common.Sm.Start, Common.Pm.Start, etc -->
  <!-- ESM does not support Start via XML configuration. Use CLI commands -->
  <Start>1</Start>
  <!-- <StartupRetries>5</StartupRetries> -->
  <!-- <StartupStableWait>10</StartupStableWait> -->

  <!-- Name, Hfi, Port, and PortGUID are ignored for ESM since they -->
  <!-- are automatically set -->
  <Name>fm0</Name> <!-- also for logging with _sm, _fe, _pm appended -->
  <Hfi>1</Hfi> <!-- local HFI to use for FM instance, 1=1st HFI -->
  <Port>1</Port> <!-- local HFI port to use for FM instance, 1=1st Port -->
  <PortGUID>0x0000000000000000</PortGUID> <!-- local port to use for FM -->
  <SubnetPrefix>0xfe80000000000000</SubnetPrefix> <!-- should be unique -->

  <!-- Overrides of the Common.Shared parameters if desired -->
  <!-- ESM does not support LogFile -->
  <!-- <LogFile>/var/log/fm0_log</LogFile> --> <!-- log for this instance -->
</Shared>

  5. Change the <SubnetPrefix> for fm0 to a unique value other than 0xfe80000000000000. A recommended value to use would be 0xfe80000000001000.

    An example of the change is shown below.

    <!-- Shared Instance config, applies to all components: SM, PM and FE -->
<Shared>
  <!-- Fm.Shared.Start controls overall startup of the Instance. -->
  <!-- If 0, none of the components in the Instance are started. -->
  <!-- If 1, instance is enabled and Fm.Sm.Start, Fm.Pm.Start, etc -->
  <!-- control startup of each manager.  The default for each manager -->
  <!-- is defined by Common.Sm.Start, Common.Pm.Start, etc -->
  <!-- ESM does not support Start via XML configuration. Use CLI commands -->
  <Start>1</Start>
  <!-- <StartupRetries>5</StartupRetries> -->
  <!-- <StartupStableWait>10</StartupStableWait> -->

  <!-- Name, Hfi, Port, and PortGUID are ignored for ESM since they -->
  <!-- are automatically set -->
  <Name>fm0</Name> <!-- also for logging with _sm, _fe, _pm appended -->
  <Hfi>1</Hfi> <!-- local HFI to use for FM instance, 1=1st HFI -->
  <Port>1</Port> <!-- local HFI port to use for FM instance, 1=1st Port -->
  <PortGUID>0x0000000000000000</PortGUID> <!-- local port to use for FM -->
  <SubnetPrefix>0xfe80000000001000</SubnetPrefix> <!-- should be unique -->

  <!-- Overrides of the Common.Shared parameters if desired -->
  <!-- ESM does not support LogFile -->
  <!-- <LogFile>/var/log/fm0_log</LogFile> --> <!-- log for this instance -->
</Shared>

  6. Find the next occurrence of <Shared>.

    An example of the shared instance for fm1 with key items in bold is shown below.

    <!-- Shared Instance config, applies to all components: SM, PM and FE -->
<Shared>
  <Start>0</Start> <!-- Overall Instance Startup, see fm0 for more info -->
  <Name>fm1</Name> <!-- also for logging with _sm, _fe, _pm appended -->
  <Hfi>1</Hfi> <!-- local HFI to use for FM instance, 1=1st HFI -->
  <Port>2</Port> <!-- local HFI port to use for FM instance, 1=1st Port -->
  <PortGUID>0x0000000000000000</PortGUID> <!-- local port to use for FM -->
  <SubnetPrefix>0xfe80000000001001</SubnetPrefix> <!-- should be unique -->
  <!-- Overrides of the Common.Shared or Fm.Shared parameters if desired -->
  <!-- <LogFile>/var/log/fm1_log</LogFile> --> <!-- log for this instance -->
  <!-- <StartupRetries>5</StartupRetries> -->
  <!-- <StartupStableWait>10</StartupStableWait> -->
</Shared>

  7. Edit fm1 settings as shown below:

    • Start = 1 (enables FM to start an instance on the second SuperNIC)

    • HFI = 2

    • Port = 1

  8. Verify that the SubnetPrefix is the second, unique subnet ID set up during software installation.

    An example of the final settings for fm1 is shown below.

    <!-- Shared Instance config, applies to all components: SM, PM and FE -->
<Shared>
  <Start>1</Start> <!-- Overall Instance Startup, see fm0 for more info -->
  <Name>fm1</Name> <!-- also for logging with _sm, _fe, _pm appended -->
  <Hfi>2</Hfi> <!-- local HFI to use for FM instance, 1=1st HFI -->
  <Port>1</Port> <!-- local HFI port to use for FM instance, 1=1st Port -->
  <PortGUID>0x0000000000000000</PortGUID> <!-- local port to use for FM -->
  <SubnetPrefix>0xfe80000000001001</SubnetPrefix> <!-- should be unique -->
  <!-- Overrides of the Common.Shared or Fm.Shared parameters if desired -->
  <!-- <LogFile>/var/log/fm1_log</LogFile> --> <!-- log for this instance -->
  <!-- <StartupRetries>5</StartupRetries> -->
  <!-- <StartupStableWait>10</StartupStableWait> -->
</Shared>

  9. Save the opafm.xml file.

  10. Copy the opafm.xml file to all secondary Fabric Manager.

  11. Restart the primary Fabric Manager using systemctl restart opafm.

  12. Restart the standby Fabric Managers using systemctl restart opafm.

  13. Run systemctl status opafm to verify that the Fabric Managers are running.