Skip to main content

Cornelis Technical Documentation

5.2.1. vFabric Configuration

vFabrics provide configuration of two properties of the fabric:

  • Partitioning

  • Quality of Service

Note

For many installations, additional vFabrics are not required and the Default vFabric in the opafm.xml file is all that is needed. No additional configuration is required. (Refer to Preconfigured vFabrics).

5.2.1.1. Considerations

When considering vFabrics, note the following:

  • If you do not need a partitioned fabric, do nothing (refer to Nonpartitioned Fabrics).

    Note

    Default and Admin vFabrics (partitions) are available by default. Refer to Preconfigured vFabrics.

  • For primary and standby Fabric Managers:

    • Ensure there is a minimum of one standby. 

      Note

      Additional standby Fabric Managers may be unnecessary and can create additional load for the primary Fabric Manager.

    • Default based on node GUID. Otherwise, configure your preference.

    • When changing another Fabric Manager configuration, always disable the standby Fabric Manager until you are finished.

    Refer to the CN5000 Fabric Installation Guide for configuring Fabric Managers.

  • For securing the Admin vFabric :

    • Define only certain nodes to be Fabric Managers with the ability to run fabric administration commands.

      This is a good practice so that anyone becoming root on a node cannot spoof a Fabric Manager or otherwise negatively impact a system.

      For more details, refer to Admin vFabric.

  • When setting QoS (instead of QOSGroups):

    • Enable one or more of the predefined vFabrics in opafm.xml (there are multiple predefined vFabrics in opafm.xml) and use QoS settings.

      For more information, refer to QoS Settings.

  • When setting up multi-tenancy:

    • Do not edit opafm.xml as it will be overwritten.

    • Define the common attributes of the fabric in opafm_pp.xml.

    • Define each tenant's vFabrics in vfs/ and dgs/.

    • Use the opafmconfigpp command to combine opafm_pp.xml (including pointers to vfs/*.xml and dgs/*.xml) with an opafm.xml, which will be used by the Fabric Manager.

    Note

    Always restart the Fabric Manager after making changes to the opafm.xml file.

    For more information, refer to Defining a Multi-Tenant Fabric and Virtual Fabrics for Multi-Tenancy.

5.2.1.1.1. Nonpartitioned Fabrics

If all the nodes (servers) are owned by a single group of users, then partitioning is probably of no interest. However, you may want to configure different QoS for different traffic types.

Configuration of this kind is done in the opafm.xml file, and many QoS strategies can be implemented simply by enabling some of the predefined vFabrics in opafm.xml.

5.2.1.1.2. Partitioned Fabrics

If you have multiple groups of nodes that are owned by different groups of users, then you will probably want to implement some kind of partitioning. An example would be a cloud service provider (CSP) who needs to allocate servers to different customers, and no customer should be able to access the servers of another customer. 

Requirements may change over time, so you need to be able to create, change, and destroy groups without disrupting the users in other groups.

The opafmvf command provides a framework for partitioned fabrics (refer to opafmvf). Fabric-wide configuration is defined in opafm_pp.xml, and configuration for the individual groups (vFabrics) is contained in files in the vfs/ and dgs/ directories. To create, change, or destroy a group, changes are made to the configuration fragments in vfs/ and dgs/, then these files are combined with opafm_pp.xml to create an opafm.xml that will be used by the Fabric Manager.

When using this method, note the following:

  • Each step of the process can be performed using the opafmvf command. 

  • Never make changes to the opafm.xml file as they will be overwritten in the combining process.

If you require QoS with multi-tenancy, use the QOSGroups feature (refer to QOSGroups Parameters) to provide different QoS settings that can be shared amongst different tenants. As the QOSGroups are fabric-wide, they are defined in opafm_pp.xml.

5.2.1.3. Securing the Default vFabric

When using a secured Default vFabric with redundant management nodes, Cornelis recommends that you explicitly specify the nodes and ports as Members of the Default vFabric. Failure to do so limits the operations that the management nodes can perform and the nodes that they can manage.

For more information, refer to Preconfigured vFabrics.

5.2.1.6. Deleting a Tenant or Storage vFabric

To delete a tenant or storage vFabric, you must first remove its members (GUIDS). The following steps provide an example.

  1. List the device groups in the /etc/opa-fm/dgs directory and locate the device group of the vFabric to delete.

    [root@rh212 dgs]# ls
opafm_dg_AdminNodes.xml  opafm_dg_Storage.xml  opafm_dg_Tenant2.xml

  2. List the contents of the device group to be removed in /etc/opa-fm/dgs and find the port GUIDs.

    [root@rh212 dgs]# cat opafm_dg_Tenant2.xml
<DeviceGroup>
    <Name>Tenant2</Name>
    <PortGUID>0x001175010174428a</PortGUID>
    <PortGUID>0x0011750101744e4e</PortGUID>
</DeviceGroup>

  3. Remove the member port GUIDs from the device group.

    [root@rh212 dgs]# opafmvf remove Tenant2 0x001175010174428a 0x0011750101744e4e
Removed ports from virtual fabric configuration 'Tenant2'

  4. Delete the tenant vFabric.

    [root@rh212 dgs]# opafmvf delete Tenant2
Deleted virtual fabric configuration 'Tenant2'
[root@rh212 dgs]# ls
opafm_dg_AdminNodes.xml  opafm_dg_Storage.xml

  5. List the contents of the vFabrics in the /etc/opa-fm/vfs directory to ensure that the vFabric has been deleted.

    [root@rh212 dgs]# ls ../vfs 
opafm_vf_Storage.xml

  6. Rebuild the opafm.xml file and reload the FM.

    [root@FM_host opa-fm]# opafmvf commit
/etc/opa-fm/opafm.xml will be overwritten!
Do you want to continue? [y/N] y
Processing files in  /etc/opa-fm/dgs
Processing files in  /etc/opa-fm/vfs
Config Check Passed!
Generated new configuration for fabric manager
[root@FM_host opa-fm]# opafmvf reload

5.2.1.7. Directing a Job or Application to Use the Desired vFabric

For more information on integrating jobs and applications, refer to Fabric Manager Integrating Job Schedulers with Virtual Fabrics and Integrating Other Service Applications with vFabrics, respectively.

  1. Run opareport -o vfinfo to locate the vFabric you want to use.

    Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Done Getting vFabric Records

vFabrics:
vFabric Index: 0   Name: Default
PKey: 0x8001   SL: 0  Select: 0x0   PktLifeTimeMult: 1
MaxMtu: unlimited  MaxRate: unlimited   Options: 0x00
QOS: Disabled  PreemptionRank: 0  HoQLife:   8 ms

vFabric
Index: 1   Name: Admin
PKey: 0x7fff   SL: 0  Select: 0x1: PKEY   PktLifeTimeMult: 1
MaxMtu: unlimited  MaxRate: unlimited   Options: 0x01: Security
QOS: Disabled  PreemptionRank: 0  HoQLife:  8 ms

2
VFs
-------------------------------------------------------------

    • For partitioning, note the PKey.

    • For QoS, note the SL.

    As an example, PKey = 0x1234 and SL = 3.

  2. Run MPI.

    When using the OPX Provider:

    • Note that OPX defaults to using PKey 1 (0x8001).

    • Set the following environment variables using the mpirun command: FI_OPX_PKEY=0x9234 FI_OPX_SL=3.

  3. Create an IPoIB interface to use.

    Normally, when you create an IB interface, it uses the vFabric with the default PKey. The default PKey is the first PKey in the PKey table, which may or may not be the vFabric named 'Default'.

    To create an IPoIB interface on a particular PKey, perform the following:

    1. Type:

      echo 0x1234 > /sys/class/net/ib0/create_child

      This device will be named ib0.1234.

    2. Configure the device with an IP address:

      ip addr add 192.168.111.74 dev ib0.1234

    Note

    To delete the interface, if necessary: 

    echo 0x1234 > /sys/class/net/ib0/delete_child