Skip to main content

Cornelis Technical Documentation

5.2.1.4. Creating vFabrics

When you create multiple vFabrics that are not used for multi-tenancy, you edit the opafm.xml file directly.

The following example shows how to set up three vFabrics by directly editing the opafm.xml file. For details on opafm.xml sections, refer to vFabric Overview.

Note

  • QoS and QOSGroups are mutually exclusive.

  • If you are using QOSGroups, create them in advance of this procedure. Refer to Creating a QOSGroup.

  1. Create three vFabrics by editing the opafm.xml file on the primary Fabric Manager with QoS and BaseSL.

    1. Open the file in a text editor.

    2. Search for /VirtualFabrics and insert a line above it.

    3. Copy the following example text and paste it on the empty line.

      Since this is an example, you can edit the text as you see fit.

      <VirtualFabric>
     <Name>VF1</Name>
     <Application>AllOthers</Application>
     <BaseSL>1</BaseSL>
     <Enable>1</Enable>
     <MaxMTU>Unlimited</MaxMTU>
     <MaxRate>Unlimited</MaxRate>
     <Member>All</Member>
     <QOS>1</QOS>
</VirtualFabric>

<VirtualFabric>
     <Name>VF2</Name>
     <Application>AllOthers</Application>
     <BaseSL>2</BaseSL>
     <Enable>1</Enable>
     <MaxMTU>Unlimited</MaxMTU>
     <MaxRate>Unlimited</MaxRate>
     <Member>All</Member>
     <QOS>1</QOS>
</VirtualFabric>

<VirtualFabric>
     <Name>VF3</Name>
     <Application>AllOthers</Application>
     <BaseSL>3</BaseSL>
     <Enable>1</Enable>
     <MaxMTU>Unlimited</MaxMTU>
     <MaxRate>Unlimited</MaxRate>
     <Member>All</Member>
     <QOS>1</QOS>
</VirtualFabric>

    For simplicity, the vFabrics are named VF1, VF2, and VF3 and correspond to the Base Service Levels (1, 2, and 3, respectively).

  2. Save the file and exit out of the editor:

  3. Copy the updated opafm.xml file to all standby servers.

  4. Restart all FMs:

    # systemctl restart opafm

    Note

    There is a specific order to stopping and starting the FM service. Refer to the CN5000 Maintenance and Troubleshooting Guide, Update the OPX Software for more information.

  5. List all of the vFabrics:

    The new vFabrics will appear in the list.

    # opapaquery -o vfList

    Note

    For details about the command in this step and the following steps, refer to the CN5000 Commands Guide.

  6. Check the information of all vFabrics.

    # opareport -o vfinfo

    Or, use this command to output similar information:

    # opasaquery -o vfinfo

  7. Get membership information for every vFabric in every node in the fabric:

    # opareport -o vfmember

  8. Get the configuration for a specific vFabric:

    # opapaquery -o vfConfig -V VF3

  9. Get the Buffer Control tables for all virtual lanes on every node in the Fabric:

    # opareport -o bfrctrl

  10. Run the following commands to view the SL to VL mappings:

    # opasaquery -o slsc
# opasaquery -o scvlt

  11. Clear the port counters before running traffic.

    Note

    For testing purposes only. Clearing the port counters is NOT recommended for systems in production.

    # opapmaquery -o clearportstatus -n 0x2 -w 0x80ff

    where (for all opapmaquery commands in this section)

    -n 0x2 corresponds to port 1

    -w 0x80ff corresponds to all virtual lands

  12. Run traffic on all three vFabrics simultaneously as follows:

    Open three terminal windows with simultaneous instances of MPI.

    Note

    The service level set in each Virtual Fabric's corresponding QoS Group determines which Virtual Fabric is used to run the command.

    The following is an example of running an IMB-Uniband benchmark (OpenMPI 4.1) as root in two hosts with 96 cores using the transport OFI-OPX.

    mpirun -np 192 --map-by ppr:96:node 
-host hds2fnc4121:96,hds2fnc4122:96 --allow-run-as-root  -mca pml cm 
-mca btl_openib_warn_no_device_params_found 0 -mca btl self,vader 
-mca mtl ofi   -x FI_PROVIDER=opx  -x FI_OPX_HFI_SELECT=0 -x FI_LOG_LEVEL=warn 
-x FI_OPX_EXPECTED_RECEIVE_ENABLE=1 --bind-to core IMB-OMPI/IMB-MPI1 Uniband 
-npmin 192 

  13. Verify that traffic is running over the different VLs by running the following command on any host:

    # opapmaquery -o getportstatus -n 0x2 -w 0x80ff

  14. Check that the port counters are incrementing:

    # opapaquery -o vfPortCounters -l 3 -P 1 -V VF1

  15. Get a sorted list of virtual fabric ports using utilization or error values (from vFabric buckets):

    # opapaquery -o vfFocusPorts -V VF2 -f 0x00030001 -S 0 -r 20

    Note

    opapaquery queries the Fabric Manager and can be run from anywhere in the fabric.

  16. View the data counters per Virtual Lane:

    # opapmaquery -o getdatacounters -n 0x2 -w 0x80ff

    Note

    By default, opapmaquery queries the PMA counters on the local host and needs to be run on each host where you want to see the counters.

  17. View error counters per Virtual Lane, if they exist.

    # opapmaquery -o geterrorcounters -n 0x2 -w 0x80ff

For details about adding, removing, and modifying vFabrics, refer to the following:

Fabric Manager - This is an OPX Software component responsible for managing the fabric using management packets over a dedicated virtual lane (VL15).

Service Level

Virtual Lane - This feature allows multiple, multiplexed, independent data to stream onto a single physical link.