Skip to main content

Cornelis Technical Documentation

5.2.10.3. Preconfigured vFabrics

Note

The opafm.xml configuration file sometimes refers to vFabrics as “partitions”. This is a term carried over from InfiniBand.

The Omni-Path Architecture requires every fabric to have a default and an admin vFabric.

5.2.10.3.1. Default vFabric

The Default vFabric is used by all end nodes to interact with the SM/SA. It includes all Devices and Applications.

The following shows an example Default vFabric.

<VirtualFabric>
    <Name>Default</Name>
    <Enable>1</Enable>
    <PKey>0x0001</PKey>
    <Security>0</Security>
    <QOS>0</QOS>
    <Member>All</Member>
    <Application>AllOthers</Application>
    <MaxMTU>Unlimited</MaxMTU>
    <MaxRate>Unlimited</MaxRate>
</VirtualFabric>

Note

It is a requirement that all chassis-managed spines (for example, SWE0) be a member of the Default vFabric so that the chassis can access and manage its own leaf switch chips.

To secure the Default vFabric, refer to Securing the Default vFabric.

5.2.10.3.2. Admin vFabric

The Admin vFabric provides limited privileges to nodes without SMs.

For the Admin vFabric, there must be an enabled vFabric with the following items:

  • A PKey of 0x7ffff (or 0xffff)

  • The SA application (for example, an Application with <Application>SA</Application>)

  • The only enabled vFabric that includes the SA application

  • Have All or AllMgmtAllowed (for example, a DeviceGroup with <Member>AllMgmtAllowed</Member>) as a Member

  • Have All (for example, a DeviceGroup with <Member>All</Member>) as a Member or LimitedMember

  • Additional Applications as needed

  • Additional DeviceGroups as Members or LimitedMembers as needed

  • All other vFabric policies (QoS, Security, MTU, etc.) may be set as needed

  • The PM application (for example, an Application with <Application>PM</Application>)

  • If the PM is enabled, the only enabled vFabric that includes the PM application

  • An application referencing the PA's ServiceID (for example, an Application with <ServiceID>0x1100d03c34845555</ServiceID>)

The following shows an example Admin vFabric.

<VirtualFabric>
    <Name>Admin</Name>
    <Enable>1</Enable>
    <PKey>0x7fff</PKey> <!-- must be OPA Management PKey -->
    <Security>1</Security>
    <QOS>0</QOS>
    <!-- <HighPriority>1</HighPriority> can make admin High priority -->
    <Member>HFIDirectConnect</Member> <!-- Both HFIs directly connected -->
    <Member>AllMgmtAllowed</Member>
    <Member>AllSWE0s</Member> <!-- so chassis CMU can access leafs & spines-->
    <!-- <Member>AdminNodes</Member> add more FF/admin nodes if desired -->
    <LimitedMember>All</LimitedMember>
    <Application>SA</Application>
    <Application>PA</Application>
    <Application>PM</Application>
    <!-- add other applications if desired -->
    <!-- <MaxMTU>2048</MaxMTU> can reduce MTU, SA only uses MADs -->
</VirtualFabric>

To secure the Admin vFabric, refer to Secure the Admin vFabric.

Subnet Manager or Subnet Management

Subnet Administrator/Administration

Partition Key - This is a unique ID assigned to an InfiniBand or vFabric partition.

Maximum Transmission Unit - This is the size of the largest protocol data unit or data packet that can be sent in a single network layer transaction.

Performance Manager/Performance Management - This is the Fabric Management entity responsible for monitoring fabric information related to the port and virtual lane level counters and the picture that they convey.

Performance Administrator/Administration - This component allows a centralized control point for querying the performance data in the fabric.