Skip to main content

Cornelis Technical Documentation

8.5.1. opareport Detailed Information

This section provides additional information about using opareport.

8.5.1.1. opareport Basics

opareport can be run with no options at all. In this mode, it provides a brief list of the nodes in the fabric, the brnodes report.

A sample of an opareport for a small fabric follows:

# opareport
Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Node Type Brief Summary

4 Connected FIs in Fabric:
NodeGUID         Type Name
    Port LID   PortGUID           Width Speed
0x00117501016a35f0 FI coyote hfi1_0
      1 0x0004 0x00117501016a35f0    4    25Gb
0x00117501016a361d FI goblin hfi1_0
      1 0x0003 0x00117501016a361d    4    25Gb
0x00117501016a365f FI ogre hfi1_0
      1 0x0005 0x00117501016a365f    4    25Gb
0x00117501016a366d FI duster hfi1_0
      1 0x0001 0x00117501016a366d    4    25Gb 
1 Connected Switches in Fabric:
NodeGUID         Type Name
    Port LID   PortGUID           Width Speed
0x00117500ff6a5619 SW edge1
      0 0x0002 0x00117500ff6a5619    1    25Gb
     12                              4    25Gb
     31                              4    25Gb
     35                              4    25Gb
     39                              4    25Gb 
1 Connected SMs in Fabric:
State       GUID               Name
Master      0x00117501016a366d duster hfi1_0

Each opareport allows for various levels of detail. Increasing detail is shown as further indentation of the additional information. The -d option to opareport controls the detail level. The default is 2. Values from 0-n are permitted. The maximum detail per report varies, but most have less than five detail levels.

Note

Several report types can include port counters if both the counters are available (via the use of the stats flag or input from a snapshot file) and a high enough detail level is used. Usually, a detail level between 5 and 8 is high enough to include per-port counters in report outputs. In addition to options already described in opareport, some reports such as errors or the use of flags such as -F linkqual already implies the use of -s.

For example, when the previous report is run at detail level 0, the output is as follows:

# opareport -d 0
Getting All Node Records...                                                     
Done Getting All Node Records                                                   
Done Getting All Link Records                                                   
Done Getting All Cable Info Records                                             
Done Getting All SM Info Records                                                
Node Type Brief Summary

4 Connected FIs in Fabric
1 Connected Switches in Fabric
1 Connected SMs in Fabric

A summary of fabric components is shown in the following example. This report is very similar to opafabricinfo. At the next level of detail, the report has more detail:

# opareport -d 1
Getting All Node Records...                                                     
Done Getting All Node Records                                                   
Done Getting All Link Records                                                   
Done Getting All Cable Info Records                                             
Done Getting All SM Info Records                                                
Node Type Brief Summary

4 Connected FIs in Fabric:
NodeGUID         Type Name
0x00117501016a35f0 FI ogre hfi1_0
0x00117501016a361d FI goblin hfi1_0
0x00117501016a365f FI coyote hfi1_0
0x00117501016a366d FI duster hfi1_0

1 Connected Switches in Fabric:
NodeGUID         Type Name
0x00117500ff6a5619 SW edge1

1 Connected SMs in Fabric:
State       GUID               Name
Master      0x00117501016a366d duster hfi1_0

The previous examples were all performed with a single report: the brnodes (Brief Nodes) report. This is just one of the many topology reports that opareport can generate.

Other reports summarize the present state of the fabric. Use these reports to analyze the configuration of the fabric and verify that the installation is consistent with the desired design and configuration. These reports include:

nodes

A more verbose form of brnode that provides much greater levels of detail to drill down into all the details of every node, even down to all the port state, IOUs/IOCs/Services, and Port counters.

comps and brcomps

Very similar to brnodes and nodes, except the reports are organized around systems. The grouping into systems is based on system image GUIDs for each node. This report presents more complex systems (such as servers with multiple SuperNICs or large switches composed of multiple switch chips).

Note

All switches implement a system image GUID and are therefore properly grouped. However, some third-party devices do not implement the system image GUID and may report a value of 0. In such a case, opareport treats each component as an independent system.

links

Presents all the links in the fabric. The output is very concise and helps to identify the connectivity between nodes in the fabric. This includes both internal (inside a large switch or system) and external ports (cables).

extlinks

Lists all the external links in the fabric, for example, those between different systems. This report omits links internal to a single system. Identification of a system is through SystemImageGuid.

lids

Similar to brnodes, however, it is organized and sorted by LID. The output is very concise and provides a simple cross reference of LIDs assigned to each SuperNIC and Switch in the fabric. This information can be useful in interpreting the output from the linear, mcast, and portusage reports.

ious

Similar to the nodes reports, however the focus is around IOUs/IOCs and IO Services in the fabric. This report identifies various IO devices in the fabric and their capabilities, such as direct-attach storage.

otherports

Lists all ports that are not connected to this fabric. This report identifies additional ports on FIs or Switches that are not connected to this fabric. For switches, these represent unused ports. For FIs, these may be ports connected to other fabrics or unused ports.

Additionally, opareport has reports that analyze the operational characteristics of the fabric and identify bottlenecks and faulty components in the fabric. These reports include:

slowlinks

Identifies links that are running slower than expected, that pinpoints bad cables or components in the fabric. For example, can identify a 4x cable that is poorly connected and only runs at 1x link width. The analysis includes both link speed and width.

slowconfiglinks

Extends the slowlinks report to also report links that have been configured (typically by software) to run at a width or speed below their potential. Includes DDR-capable links that have been forced to run at SDR rates.

slowconnlinks

Extends on the slowconfiglinks report to also report links that are cabled such that one of the ends of the link can never run to its potential. For example, a DDR-capable SuperNIC connected to an SDR switch.

misconfiglinks

Similar to slowconfiglinks in that it reports links that have been configured to run below their potential. However, the report does not include links that are running slower than expected.

misconnlinks

Similar to slowconnlinks in that it reports links that have been connected between ports of different speed potential. However, the report does not include links that are running slower than expected, nor links that have been configured to run slower than their potential.

errors

Performs a single point-in-time analysis of the PMA port counters for every node and port in the fabric. All the counters are compared against configured thresholds. Defaults are listed in the opamon.conf file. Any link whose counters exceed these thresholds are listed. Depending on the detail level, the exact counter and threshold are reported. This is a powerful way to identify marginal links in the fabric such as bad or loose cables or damaged components. The opamon.si.conf file can also be used to check for any non-zero values for signal integrity (SI) counters.

route

Identifies two end points in the fabric (by node name, node GUID, port name, port GUID, system image GUID, LID, port GID, IOC GUID, or IOC name), and obtains a list of all the links and components used when these two end points communicate. If there are multiple paths between the end points, such as an FI with 2 connected ports or a system with 2 FIs, the route for every available path is reported based on presently configured routing tables.

linear

Shows the linear forwarding table for each switch in the fabric. Used to manually review the routing of unicast traffic in the fabric. For each switch, every unicast LID is shown along with the port it is routed out (egress port), and the neighboring Node and Port. For large fabrics, this report can be quite large.

mcast

Shows the multicast forwarding table for each switch in the fabric. Used to manually review the routing of multicast traffic in the fabric. For each switch, every multicast LID is shown along with the list of ports it is routed out. For large fabrics, this report can be quite large.

portusage

Provides a summary analysis of the unicast routing in the fabric, in terms of how many LIDs of each node type are routed out a given port. Used for analysis of how balanced the routes in the fabric are, especially for ISLs and core switches. For each switch, all the ports are shown along with the counts of how many unicast LIDs are routed out each port. The total is shown along with SuperNIC-All, SuperNIC-Base, and Switch.

  • HFI-All includes all LIDs that correspond to a SuperNIC, including LIDs that are the base LID of the SuperNIC and LIDs that map to the SuperNIC through LMC masking.

  • HFI-Base includes only LIDs that correspond to the base LID of a SuperNIC. HFI-Base is always a subset of HFI-All.

  • Switch includes all LIDs that correspond to a Switch.

pathusage

Computes all the FI-to-FI dLID paths through the fabric and reports on the usage of each ISL Port (Switch-to-Switch link). The -F option indicates the switches and the ports on those switches to analyze. Switch Port 0 is always omitted from the analysis. These reports can also be run against snapshots that were performed with the -r option.

treepathusage

Similar to pathusage with the exception that treepathusage is applicable only to Fat Tree topologies and provides specific analysis of uplink and downlink paths, indicating what tier each switch is in within the fabric.

8.5.1.2. Simple Topology Verification

opareport provides a flexible way to identify changes to the fabric or the appropriate reassembly of the fabric after a move. For example, run opareport after staging and testing the fabric in a remote location before final installation at a customer site.

This type of report can be saved for later comparison to a future report. Since opareport produces simple text reports, standard tools such as sdiff (side by side diff) can be used for comparison and analysis of the changes.

In this mode of operation, all previous reports are available, however, you can filter the information that is output. Use the all report to include all reports of general interest.

Use the -P option to omit information that does not persist across a fabric reboot, for example, LIDs and error counters. In the report, the information is marked out with xxx.

If software configuration changes are anticipated, use the opareport -H option to only include hardware information. Use this option when adjusting the timeouts the SM configures in the fabric.

Use the -N option to omit all the node and IOC names from the report. If changes are anticipated in this area, this option can be used so future differences do not report changes in names.

8.5.1.3. Advanced Topology Verification

You can use the -T option for opareport to compare the state of the fabric against a previous state or a user-generated configuration for the fabric.

The XML description used by the -T option is the same as the XML format generated by the -o links or -o extlinks and/or -o brnodes reports when they are run with the -x option. The opareport -o topology argument is an easy way to generate such a report and is equivalent to specifying all three of these reports.

A simple way to perform topology verification against a previous configuration is to generate the previous topology using a command such as:

opareport -o topology -x > topology.xml

Later, the fabric can be compared against that topology using a command such as:

opareport -T topology.xml -o verifyall

Unlike simple diff comparisons, this method of topology verification performs a more context-sensitive comparison and presents information in terms of links, nodes, or SMs that are missing, unexpected, or incorrectly configured.

All the other capabilities of opareport are fully available when using a topology_input file. For example, snapshot_input files can also be used to generate or compare topologies based on previous fabric snapshots. In addition, the -F option may be used to focus the analysis.

Note

verify* reports may still report missing links, nodes, or SMs outside the scope of the desired focus.

There are multiple variations of advanced topology verification: verifycas, verifysws, verifyrtrs, verifysms, verifylinks, and verifyextlinks. In addition, verifynodes and verifyall can be used to generate combined reports.

verifylinks and verifyextlinks perform the same analysis, however, they differ in the scope of the analysis. verifylinks checks all links in the fabric. In contrast, verifyextlinks performs the following:

  • Limits its verification to links outside of a system.

  • Does not analyze links between nodes with the same SystemImageGuid, such as within a large CN5000 Omni-Path Chassis.

  • Ignores links from the topology_input file that specify a non-zero value for the XML tag <Internal> within the <Link> tag.

The XML format of topology_input file is shown in the following example. The example is purposely brief and omits many links, nodes, and SMs.

<?xml version="1.0" encoding="utf-8" ?>
<Report>
<LinkSummary>
<Link>
<Rate>25g</Rate>
<MTU>8192</MTU>
<Internal>0</Internal>
<LinkDetails>SampleHost1 to Switch</LinkDetails>
<Cable>
<CableLength>11m</CableLength>
<CableLabel>S4567</CableLabel>
<CableDetails>sample cable model xxx</CableDetails>
</Cable>
<Port>
<NodeGUID>0x0x0011750101660572</NodeGUID>
<PortGUID>0x0011750101660572</PortGUID>
<PortNum>1</PortNum>
<NodeType>FI</NodeType>
<NodeDesc>SampleHost1 HFI-1</NodeDesc>
<PortDetails>SampleHost1 primary port</PortDetails>
</Port>
<Port>
<NodeGUID>0x0011750007000df6</NodeGUID>
<PortNum>1</PortNum>
<NodeType>SW</NodeType>
<NodeDesc>SampleSwitch1 Leaf 4, Chip A</NodeDesc>
</Port>
</Link>
<Link>
<Rate>25g</Rate>
<MTU>8192</MTU>
<Internal>0</Internal>
<Port>
<NodeGUID>0x0011750101660574</NodeGUID>
<PortGUID>0x0011750101660574</PortGUID>
<PortNum>1</PortNum>
<NodeType>FI</NodeType>
<NodeDesc>SampleHost2 HFI-1</NodeDesc>
</Port>
<Port>
<NodeGUID>0x0011750007000e6d</NodeGUID>
<PortNum>4</PortNum>
<NodeType>SW</NodeType>
<NodeDesc>SampleSwitch1 Leaf 5, Chip A</NodeDesc>
</Port>
</Link>
</LinkSummary>
<Nodes>
<FIs>
<Node id="0x0011750101660576">
<NodeGUID>0x0011750101660576</NodeGUID>
<NodeDesc>SampleHost2 HFI-1</NodeDesc>
<NodeDetails>SampleHost2 only HFI</NodeDetails>
</Node>
</FIs>
<Switches>
<Node id="0x001175000600025a">
<NodeGUID>0x001175000600025a</NodeGUID>
<NodeDesc>SampleSwitch1 Spine 1, Chip A</NodeDesc>
<NodeDetails>core switch</NodeDetails>
</Node>
</Switches>
<SMs>
<SM id="0x0011750101660578:1">
<NodeGUID>0x0011750101660578</NodeGUID>
<NodeDesc>SampleHost2 HFI-1</NodeDesc>
<PortNum>1</PortNum>
<PortGUID>0x0011750101660579</PortGUID>
<NodeType>FI</NodeType>
<NodeType_Int>1</NodeType_Int>
<SMDetails>SampleHost2 SM</SMDetails>
</SM>
</SMs>
</Nodes>
</Report>

The XML tags have the following meanings:

<Report>

Primary top level tag. Exactly one such tag is permitted per file. Alternatively, this may be <Topology>.

<LinkSummary>

Container tag describing all the links expected in the fabric. Alternatively, <ExternalLinkSummary> may be used. <ExternalLinkSummary> should be used if the file only describes external links. If both external and internal links are described, <LinkSummary> should be used. Only one of these two choices is permitted per file.

<Link>

Container tag describing a single link. Many instances of this tag can occur per <LinkSummary> or <ExternalLinkSummary>.

<Link> allows the following tags:

<Rate>

String describing the expected rate of the link. Valid values are 2.5g, 5g, 10g, 20g, 30g, 40g, 60g, 80g, or 120g. The value is case-insensitive but must contain no extra whitespace. Alternatively, an integer value <Rate_Int> may be provided based on the values for Rate from the SMA packets. If both <Rate> and <Rate_Int> are specified, whichever value appears later within the given link is used. If neither is specified, the rate of the link is not verified.

<MTU>

An integer describing the expected MTU of the link. Valid values are 256, 512, 1024, 2048, and 4096. If not specified, the MTU of the link is not verified.

<Internal>

A flag indicating if the link is internal or external. A value of 0 indicates external links that are processed by both verifylinks and verifyextlinks. A value of 1 indicates an internal link that is only processed by verifylinks. If omitted, the actual fabric link attributes or the attributes of the port are used to determine if the link should be processed. The value for this field is not verified against the actual fabric.

<LinkDetails>

A free-form text field of up to 64 characters. This field is optional. When provided, this is output as a link attribute in all reports that show link details, such as links, extlinks, route, verifylinks, and verifyextlinks reports. Cornelis recommends you use this field to describe the purpose of the link. This field can also be used by the linkdetpat focus option to select the link.

<Cable>

A container tag providing additional information about the cable.

<Cable> allows the following tags:

<CableLength>

A free-form text field of up to 10 characters. This field is optional. When provided, this is output as a link cable attribute in all reports that show link details, such as links, extlinks, route, verifylinks, and verifyextlinks reports. Cornelis recommends you use this field to describe the length of the cable using text such as 11m. This field can also be used by the lengthpat focus option to select the link.

<CableLabel>

A free-form text field of up to 20 characters. This field is optional. When provided, this is output as a link cable attribute in all reports that show link details, such as links, extlinks, route, verifylinks, and verifyextlinks reports. Cornelis recommends you use this field to describe the identifying label attached to the cable using text such as S4576. This field can also be used by the labelpat focus option to select the link. Using this field to match the actual unique physical labels placed on the cables during installation can greatly help cross-referencing the reports to the physical cluster, such as when needing to identify or replace cables.

<CableDetails>

A free-form text field of up to 64 characters. This field is optional. When provided, this is output as a link attribute in all reports that show link details, such as links, extlinks, route, verifylinks, and verifyextlinks reports. Cornelis recommends you use this field to describe the type, model, and/or manufacturer of the cable. This field can also be used by the cabledetpat focus option to select the link.

<Port>

A container tag providing additional information about the two ports that make up the link.

<Port> allows the following tags:

<NodeGUID>

Node GUID reported by the SMA for the given FI or switch.

<PortGUID>

Port GUID reported by the SMA for the given FI or switch.

Note

Switches only have PortGuids for port 0 (the internal management port), while FIs have a unique GUID for every port.

<PortNum>

Port Number within the FI or switch.

<NodeDesc>

Node Description reported by the FI or switch. Cornelis recommends that you configure a unique value for this field in each node in your fabric. For example, CN5000 OPX Software Linux hosts use the combination of Linux hostname and SuperNIC number to create a unique NodeDesc.

<NodeType>

Node type reported by the node. Values include: FI or SW. Alternatively, an integer value <NodeType_Int> may be provided based on the values for NodeType from the SMA packets. If both <NodeType> and <NodeType_Int> are specified, whichever appears later within the given Port is used. If neither is specified, the node type of the port is not verified.

<PortDetails>

Free-form text field of up to 64 characters. This field is optional. When provided, this is output as a port attribute in all reports that show port details, such as links, extlinks, route, comps, verifylinks, and verifyextlinks reports. Cornelis recommends you use this field to describe the purpose of the port. This field can also be used by the portdetpat focus option to select the port.

The previous fields are used to associate a port in the topology_input file with an actual port in the fabric, also called resolving the port. You need not provide all of the information. Association to an actual port in the fabric is performed using the following order of checks based on the tags that are specified:

  • NodeGUID, PortNum

  • NodeGUID, PortGUID

  • NodeGUID – if given FI has exactly 1 port.

  • NodeDesc, PortNum

  • NodeDesc, PortGUID

  • NodeDesc – if given FI has exactly 1 port.

  • PortGUID, PortNum – useful to select ports other than 0 on a switch.

  • PortGUID

If NodeDesc is used to specify ports, it is important that the fabric is configured such that each NodeDesc is unique. Otherwise, the <Port> may resolve to a different port than desired, which could result in incorrect results or errors during topology verification.

When redundant information is provided, the extra information is ignored while resolving the port. However, during verifylinks or verifyextlinks all the input provided is verified against the actual fabric and any discrepancies are reported.

Some examples of redundant information:

  • NodeGuid, NodeDesc – NodeDesc is not used to resolve port.

  • NodeGuid, PortNum, PortGuid – PortGuid is not used to resolve port.

  • NodeDesc, PortNum, PortGuid – PortGuid is not used to resolve port.

The <NodeType> field is never used during resolution; it is only used during verification.

<Nodes>

Container tag describing all the nodes expected in the fabric.

<FIs>

Container tag describing all the FIs expected in the fabric. Many instances of this tag can occur per <Nodes>.

<Switches>

Container tag describing all the Switches expected in the fabric. Many instances of this tag can occur per <Nodes>.

<SMs>

Container tag describing all the SMs expected in the fabric. Many instances of this tag can occur per <Nodes>.

<Node>

Container tag describing a single node (FI or SW). Many instances of this tag can occur per <FIs> or <Switches>.

<Node> allows the following tags:

<NodeGUID>

Node GUID reported by the SMA for the given FI or Switch.

<NodeDesc>

Node Description reported by the FI or switch. Cornelis recommends that you configure a unique value for this field in each node in your fabric. For example, CN5000 OPX Software Linux hosts use the combination of Linux hostname and SuperNIC number to create a unique NodeDesc.

<NodeDetails>

Free-form text field of up to 64 characters. This field is optional. When provided, this is output as a node attribute in all reports that show node details, such as links, extlinks, route, comps, verifycas, verifysws, verifyrts, verifylinks, and verifyextlinks reports. Cornelis recommends you use this field to describe the purpose and/or model of the node. This field can also be used by the nodedetpat focus option to select the node.

The previous fields are used to associate a Node (FI or Switch) in the topology_input file with an actual node in the fabric, also called resolving the node. You need not provide all of the information. Association to an actual node in the fabric is performed using the following order of checks based on the tags that are specified:

  • NodeGUID

  • NodeDesc

If NodeDesc is used to specify nodes, the fabric must be configured such that each NodeDesc is unique. Otherwise, the <Node> may resolve to a different node than desired, which could result in incorrect results or errors during topology verification.

When redundant information is provided, the extra information is ignored while resolving the node. However, during verifycas, verifysws, or verifyrtrs, all the input provided is verified against the actual fabric and any discrepancies are reported.

An example of redundant information:

  • NodeGuid, NodeDesc - NodeDesc is not used to resolve node.

The node type (as implied by the container tag for the <Node>) is never used during resolution, it is only used during verification.

<SM>

Container tag describing a single SM. Many instances of this tag can occur per <SMs>.

<SM> allows the following tags:

<NodeGUID>

Node GUID reported by the SMA for the given FI or switch that is running the FM.

<NodeDesc>

Node Description reported by the FI or switch that is running the SM. Cornelis recommends that you configure a unique value for this field in each node in your fabric. For example,CN5000 OPX Software Linux hosts use the combination of Linux hostname and SuperNIC number to create a unique NodeDesc.

<PortGUID>

Port GUID reported by the SMA for the given FI or switch that is running the SM.

Note

Switches only have PortGuids for port 0 (the internal management port), while FIs have a unique GUID for every port.

<PortNum>

Port Number within the FI or switch that is running the SM.

<NodeType>

Node type reported by the node that is running the SM. Values include: FI or SW. Alternatively, an integer value <NodeType_Int> may be provided based on the values for NodeType from the SMA packets. If both <NodeType> and <NodeType_Int> are specified, whichever appears later within the given port is used. If neither is specified, the node type of the SM is not verified.

<SMDetails>

Free-form text field of up to 64 characters. This field is optional. When provided, this is output as an SM attribute in all reports that show SM details, such as comps and verifysms reports. Cornelis recommends you use this field to describe the purpose of the SM. This field can also be used by the smdetpat focus option to select the SM.

The previous fields are used to associate a port running an SM in the topology_input file with an actual port in the fabric, also called resolving the SM. You need not provide all of the information. Association to an actual port in the fabric is performed using the following order of checks based on the tags that are specified:

  • NodeGUID, PortNum

  • NodeGUID, PortGUID

  • NodeGUID – if given FI has exactly 1 active port or is a switch.

  • NodeDesc, PortNum

  • NodeDesc, PortGUID

  • NodeDesc – if given FI has exactly 1 active port or is a switch.

  • PortGUID, PortNum – limited usefulness.

  • PortGUID

If NodeDesc is used to specify SM ports, the fabric must be configured such that each NodeDesc is unique. Otherwise, the <SM> may resolve to a different port than desired, which could result in incorrect results or errors during topology verification.

When redundant information is provided, the extra information is ignored while resolving the port for an SM. However, during verifysms all the input provided is verified against the actual fabric and any discrepancies are reported.

Some examples of redundant information:

  • NodeGuid, NodeDesc – NodeDesc is not used to resolve port.

  • NodeGuid, PortNum, PortGuid – PortGuid is not used to resolve port.

  • NodeDesc, PortNum, PortGuid – PortGuid is not used to resolve port.

The NodeType field is never used during resolution, it is only used during verification.

8.5.1.4. Augmented Report Information

A topology_input file includes additional information including cable (length, label, details), links (details), ports (details), nodes (details), and SMs (details). The file can be used during any report to provide information about the fabric that is not electronically available. This can help cross-reference the output of the report against the physical fabric. For example, if the cable length field is supplied, reports can be focused on all cables of a given length. Similarly, if cable labels are supplied, the report output includes the labels, making it much easier to locate the actual cables for tasks such as rerouting or replacement.

8.5.1.5. Focused Reports

One of the more powerful features of opareport is the ability to focus a report on a subset of the fabric. Using the -F option, you can specify a node name, node name pattern, node GUID, node type, port GUID, IOC name, IOC name pattern, IOC GUID, IOC type, system image GUID, port GID, port rate, port state, port physical state, MTU capability, LID, link quality indicator, cable info for cable length, cable info for vendor name, cable info for vendor part number, cable info for vendor rev, cable info for vendor serial number, or SM.

The subsequent report indicates the total components in the fabric but only reports on those that relate to the focus area. For example, in a nodes report, if a port is specified for focus, only the node containing that port is reported on. In a links report, if a port is specified for focus, only the link using that port is reported.

When a focus is used for fabric analysis, -o errors, -C or -i, the analysis includes all the ports selected by the focus as well as their neighbors. If desired, the -L option limits the operation to exactly the selected ports.

You may choose a focus level that is different from the orientation of the report. For example, if a node name is specified as the focus for a links report, a report of all the links to that node is provided. This includes multiple switch ports or FI ports.

You can perform reverse lookups by carefully using this feature of report focus. For example, requesting a brnodes report with a focus on a LID performs reverse lookup on that LID and indicates what node it is for.

When focusing a report, you can also specify a detail level. For detail 0, the report shows only a count of number of matches. For detail 1, the report shows only the highest level of the entity that matches.

8.5.1.6. Advanced Focus

As mentioned previously, you can focus a report on a subset of the fabric. In addition, you can further limit the report focus using the following methods.

The beginning of a focused report includes a summary of the items focused on. When the focus has a large scope, this list can be quite long. To omit the summary section from the report, use the -Q option.

  • Port number specifier

    The node name, node name pattern, node GUID, node type, IOC name, IOC name pattern, IOC GUID, IOC type, and system image GUID also allow for a port number specifier. This limits the focus to the given port number. If the selection resolves to multiple switches or FIs, all ports on the present fabric matching the given port number are selected for the report. For example, in a system composed of multiple nodes, there may be multiple ports with the same port number.

  • Route between points

    This method focuses on all the ports involved in a particular route and can be an excellent way to determine a performance or error situation reported between two specific points in the fabric. For example, MPI may report StatusTimeoutRetry between two processes in its run.

    * syadmin fields supplied in a topology file, typically generated by opaxlattopology, including cable labels, cable details, planned cable length, link details, port details, and SM details.

  • glob-style patterns

    You can use a wildcard focus for the node name, IOC name, node details, cable label, cable length, cable details, cable vendor name, cable vendor part number, cable vendor rev, cable vendor serial number, link details, port details, or SM details. If a consistent naming convention is used for fabric components, this method provides a powerful way to focus reports on nodes. If the host names are prefixed with an indication of their purpose, searches can be performed based on the purpose of the node.

    For example, if you use a naming convention such as the following: l### = login node ###, n### = compute node ###, s### = storage node ###, then you can create a report using one of the following patterns: 'l*', 'n*', or 's*'.

    Note

    A glob style pattern is a shell-style wildcard pattern as used by bash and other tools. If you use this style of pattern, you must also use single quotes so the shell does not try to expand them to match local file names.

8.5.1.7. Focus Examples

Examples of using the focus options are shown in the following list:

opareport -o nodes -F portguid:0x00117500a000447b
opareport -o nodes -F nodeguid:0x001175009800447b:port:1
opareport -o nodes -F nodeguid:0x001175009800447b
opareport -o nodes -F node:duster
opareport -o nodes -F node:duster:port:1
opareport -o nodes -F 'nodepat:d*'
opareport -o nodes -F 'nodepat:d*:port:1'
opareport -o nodes -F nodetype:FI
opareport -o nodes -F nodetype:FI:port:1
opareport -o nodes -F lid:1
opareport -o nodes -F lid:1:node
opareport -o nodes -F gid:0xfe80000000000000:0x00117500a000447b
opareport -o nodes -F systemguid:0x001175009800447b
opareport -o nodes -F systemguid:0x001175009800447b:port:1
opareport -o nodes -F iocguid:0x00117501300001e0
opareport -o nodes -F iocguid:0x00117501300001e0:port:2
opareport -o nodes -F 'ioc:Chassis 0x001175005000010C, Slot 2, IOC 1'
opareport -o nodes -F 'ioc:Chassis 0x001175005000010C, Slot 2, IOC 1:port:2'
opareport -o nodes -F 'iocpat:*Slot 2*'
opareport -o nodes -F 'iocpat:*Slot 2*:port:2'
opareport -o nodes -F ioctype:XXXX
opareport -o nodes -F ioctype:XXXX:port:2
opareport -o nodes -F sm
opareport -o nodes -F route:node:duster:node:cuda
opareport -o nodes -F route:node:duster:port:1:node:cuda:port:2

8.5.1.8. Scriptable Output

opareport permits custom scripting. As previously mentioned, options like -H, -P, and -N generate reports that can be compared to each other. The -x option permits output reports to be generated in XML format. The XML hierarchy is similar to the text-based reports. Using XML permits other XML tools (such as PERL XML extensions) to easily parse opareport output, enabling you to create scripts to further search and refine report output formats.

The opaxmlextract tool easily converts between XML files and delimited text files.

You can integrate opareport into custom scripts. You can also generate customer-specific new report formats and cross-reference opareport with other site-specific information.

8.5.1.9. Monitor for Fabric Changes Using opareport

opareport can easily be used in other scripts. For example, the following simple script can be run as a cron job to identify if the fabric has changed from the initial design.

#!/bin/bash
# specify some filenames to use
expected_config=/usr/local/report.master # master copy of config previously created
config=/tmp/report$$ # where we will generate new report
diffs=/tmp/report.diff$$ # where we will generate diffs 
opareport -o all -d 5 -P > $config 2>/dev/null
if ! diff $config $expected_config > $diffs 2>/dev/null
then
# notify admin, for example mail the new report to the admin
 cat $diffs $expected_config $config |
mail -s "fabric change detected" admin@somewhere
fi
rm -f $config $diffs

8.5.1.10. Sample Outputs

8.5.1.10.1. Analyze all ports in fabric for errors, inconsistent connections, bad cables
[root@duster root]# opareport -o errors -o slowlinks
Links running slower than expected Summary

Links running slower than expected:
Rate NodeGUID          Port Type Name
     Active                            Enabled
     Lanes, Used(Tx), Used(Rx), Rate,  Lanes,   DownTo,  Rates
100g 0x00117501025019ab  44 SW   edge1
     4      3         3         25Gb   1,2,3,4  3,4      25Gb        
<->  0x0011750102513139  44 SW   edge2
     4      3         3         25Gb   1,2,3,4  3,4      25Gb        
100g 0x00117501025019ab  48 SW   edge1
     4      3         4         25Gb   1,2,3,4  3,4      25Gb        
<->  0x0011750102513139  48 SW   edge2
     4      4         3         25Gb   1,2,3,4  3,4      25Gb        
96 of 96 Links Checked, 2 Errors found
-------------------------------------------------------------------------------
Links with errors > threshold Summary

Configured Error Thresholds:
    LinkQualityIndicator           5
    LinkDowned                     3
    RcvErrors                      100
    ExcessiveBufferOverruns        3
    LinkErrorRecovery              3
    LocalLinkIntegrityErrors       3
    XmitConstraintErrors           10
    RcvConstraintErrors            10
    CongDiscards                   100
96 of 96 Links Checked, 0 Errors found
-------------------------------------------------------------------------------
8.5.1.10.2. Identify the route between two nodes
[root@goblin root]#   opareport -o route -S node:"goblin hfi1_0" -D node:"orc hfi1_0"
Getting All Node Records...                                                     
Done Getting All Node Records                                                   
Done Getting All Link Records                                                   
Done Getting All Cable Info Records                                             
Done Getting All SM Info Records                                                
Routes Summary Between:
    Node: 0x001175010157409d FI goblin hfi1_0
and Node: 0x001175010157403d FI orc hfi1_0

Routes between ports:
     0x001175010157409d   1 FI   goblin hfi1_0
and  0x001175010157403d   1 FI   orc hfi1_0
2 Paths
    SGID: 0xfe80000000000000:001175010157409d
    DGID: 0xfe80000000000000:001175010157403d
    SLID: 0x0001 DLID: 0x0018 Reversible: Y PKey: 0x8001
    Raw: N FlowLabel: 0x00000 HopLimit: 0x00 TClass: 0x00
    SL:  0 Mtu:  8192 Rate: 100g PktLifeTime:  134 ms Pref: 0
        Rate NodeGUID          Port Type Name
        100g 0x001175010157409d   1 FI   goblin hfi1_0
        ->   0x00117501025131cb  44 SW   edge1
        100g 0x00117501025131cb  40 SW   edge2
        ->   0x001175010157403d   1 FI   orc hfi1_0
    2 Links Traversed
    SGID: 0xfe80000000000000:001175010157409d
    DGID: 0xfe80000000000000:001175010157403d
    SLID: 0x0001 DLID: 0x0018 Reversible: Y PKey: 0xffff
    Raw: N FlowLabel: 0x00000 HopLimit: 0x00 TClass: 0x00
    SL:  0 Mtu:  8192 Rate: 100g PktLifeTime:  134 ms Pref: 0
        Rate NodeGUID          Port Type Name
        100g 0x001175010157409d   1 FI   goblin hfi1_0
        ->   0x00117501025131cb  44 SW   edge1
        100g 0x00117501025131cb  40 SW   edge1
        ->   0x001175010157403d   1 FI   orc hfi1_0
    2 Links Traversed
8.5.1.10.3. Obtain very detailed information about nodes

Note

To shorten the length of the output, the following example focuses on only one node.

[root@phwtpriv27 ~]$ opareport -o nodes -F node:"duster hfi1_0" -d 5 -s
Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Done Getting vFabric Records
Getting All Port Counters...
Done Getting All Port Counters
Node Type Summary
Focused on:
    Node: 0x0002b3010100002b FI duster hfi1_0

48 Connected FIs in Fabric:
    Name: duster hfi1_0
        NodeGUID: 0x0002b3010100002b Type: FI
        Ports: 1 PartitionCap: 16 SystemImageGuid: 0x0002b3010100002b
        BaseVer: 128 SmaVer: 128 VendorID: 0x2b3 DeviceID: 0x7220 Rev: 0x60002
    1 Connected Ports:
        PortNum:   1 LID: 0x0006 GUID: 0x0002b3010100002b
            Neighbor:  Name: MOOSE_STL_SWITCH0
                NodeGUID: 0x0002b30102000000 Type: SW PortNum:   5
            LocalPort: 1   PortState: Active                 PhysState: LinkUp
            IsSMConfigurationStarted: True  NeighborNormal: True
            PortType: Standard
            LID:    0x0006              LMC: 0               Subnet: 0xfe80000000000000
            SMLID:  0x0002   SMSL:  0   RespTimeout:    8 ms  SubnetTimeout:  536 ms
            M_KEY:  0x0000000000000000  Lease:     0 s          Protect: Read-only
            MTU  Supported: (0x6) 8192 bytes
            VLStallCount (per VL):  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            MTU  Active by VL:
            00: 8192 01:    0 02:    0 03:    0 04:    0 05:    0 06:    0 07:    0
            08:    0 09:    0 10:    0 11:    0 12:    0 13:    0 14:    0 15: 2048
            16:    0 17:    0 18:    0 19:    0 20:    0 21:    0 22:    0 23:    0
            24:    0 25:    0 26:    0 27:    0 28:    0 29:    0 30:    0 31:    0
            LinkWidth: Active:       4  Supported:    1,2,3,4  Enabled:    1,2,3,4
            LinkWidthDnGrade: ActiveTx:  4 Rx:  4 Supported:    1,2,3,4  Enabled:        3,4
            PortLinkMode: Active: STL      Supported: STL              Enabled: STL
            LinkSpeed: Active:    25Gb  Supported: 25Gb  Enabled: 25Gb
            SM_TrapQP: 0x0  SA_QP: 0x1 IPAddr Prim/Sec: :: / 0.0.0.0
            VLs:       Active:     8+1  Supported:        8+1
                HOQLife (Per VL):
                VL 0: 0x0 VL 1: 0x0 VL 2: 0x0 VL 3: 0x0 VL 4: 0x0
                VL 5: 0x0 VL 6: 0x0 VL 7: 0x0 VL 8: 0x0 VL 9: 0x0
                VL10: 0x0 VL11: 0x0 VL12: 0x0 VL13: 0x0 VL14: 0x0
                VL15: 0x0

            VL Arb Cap:  High:      16        Low:         16  HiLimit:    0 PreemptLimit    0
            VLFlowControlDisabledMask: 0x00000000
            NeighborMode  MgmtAllowed:  No  FwAuthenBypass: Off  NeighborNodeType: Switch
            Capability 0x00000000: -
            Capability3 0x0008: SS
            Violations: M_Key:     0 P_Key:     0 Q_Key:     0
            PortMode ActiveOptimize: Off PassThrough: Off VLMarker: Off
            FlitCtrlInterleave Distance Max:  0 Enabled:  0
            MaxNestLevelTxEnabled: 10 MaxNestLevelRxSupported: 10
            SmallPktLimit: 0x07 MaxSmallPktLimit: 0xff PreemptionLimit: 0x10
            FlitCtrlPreemption MinInital: 80 MinTail: 80 LargePktLim: 0x07
            BufferUnits: VL15Init 0x0040; VL15CreditRate 0x00; CreditAck 0x0; BufferAlloc 0x3
            PortErrorActions: 0x172000: CE-UVLMCE-BCDCE-BTDCE-BHDR-BVLM
            ReplayDepth Buffer 0x00; Wire 0x00
            DiagCode: 0x0000
            OverallBufferSpace: 0x0900
            P_Key Enforcement: In: Off Out: Off
            Performance: Transmit
                Xmit Data                      0 MB (5677 Flits)
                Xmit Pkts                     63
                MC Xmt Pkts                    0
            Performance: Receive
                Rcv Data                       0 MB (3762 Flits)
                Rcv Pkts                      57
                MC Rcv Pkts                    0
            Performance: Congestion
                Congestion Discards                      0
                Rcv FECN                                 0
                Rcv BECN                                 0
                Mark FECN                                0
                Xmit Time Congestion                     0
                Xmit Wait                                0
            Performance: Bubbles
                Xmit Wasted BW                           0
                Xmit Wait Data                           0
                Rcv Bubble                               0
            Errors: Signal Integrity
            Link Qual Indicator                          5 (Excellent)
                Uncorrectable Errors                     0
                Link Downed                              0
                Rcv Errors                               0
                Exc. Buffer Overrun                      0
                FM Config Errors                         0
                Link Error Recovery                      0
                Local Link Integ Err                     0
                Rcv Rmt Phys Err                         0
            Errors: Security
                Xmit Constraint                          0
                Rcv Constraint                           0
            Errors: Routing
                Rcv Sw Relay Err                         0
                Xmit Discards                            0
            QSFP: ActiveCu , 1m   STL Simulator
                  Power Class 5, 4.0W max S/N 0x0000000000033
                  OUI 0x0002B3
                  Cable Type: Linear active copper cable
                  Speed Sup: 0 Gb
1 Matching FIs Found

1 Connected Switches in Fabric:
0 Matching Switches Found

1 Connected SMs in Fabric:
0 Matching SMs Found

-------------------------------------------------------------------------------
8.5.1.10.4. Obtain very detailed information about IOCs

Note

To shorten the length of the output, the following example focuses on only one IOC.

[root@duster root]# opareport -o ious -F ioc:'Chassis 0x00066A005000010C, Slot 2, 
IOC 2' -d 5
IOU Summary Focused on:
    Ioc:   2 0x00066a02300001e0 Chassis 0x00066A005000010C, Slot 2, IOC 2
        in Node: 0x00066a00580001e0 FI VEx in Chassis 0x00066a005000010c, Slot 
1 IOUs in Fabric:
    Name: VEx in Chassis 0x00066a005000010c, Slot 2
        NodeGUID: 0x00066a00580001e0 Type: FI
        Ports: 2 PartitionCap: 64 SystemImageGuid: 0x00066a00580001e0
        BaseVer: 1 SmaVer: 1 VendorID: 0x66a DeviceID: 0x5a44 Rev: 0xa1
    1 Connected Ports:
        PortNum:   2 LID: 0x0013 GUID: 0x00066a02580001e0
            Neighbor:  0x00066a00280002cd   3 SW InfiniCon Systems InfiniFabric
(Sw A Dev A)
            PortState: Active           PhysState: LinkUp    DownDefault: Pollig
            LID:    0x0013              LMC: 0               Subnet: 0xfe8000000000000
            SMLID:  0x0001   SMSL:  0   RespTimeout:   33 ms  SubnetTimeout:  56 ms
            M_KEY:  0x0000000000000000  Lease:     0 s       Protect: Readonly
            MTU:       Active:    2048  Supported:    2048  VL Stall: 0
            LinkWidth: Active:      4x  Supported:    1-4x  Enabled:    1-4x
            LinkSpeed: Active:   2.5Gb  Supported:   2.5Gb  Enabled:   2.5Gb
            VLs:       Active:     1+1  Supported:     4+1  HOQLife: 4096 ns
            Capability 0x02090048: CR DM CM SL Trap
            Violations: M_Key:     0 P_Key:     0 Q_Key:     0
            ErrorLimits: Overrun: 15 LocalPhys: 15  DiagCode: 0x0000
            P_Key Enforcement: In: Off Out: Off  FilterRaw: In: Off Out: Off
    Max IOCs:   3 Change ID:     9 DiagDeviceId: 0 Rom: 0
        IocSlot:   2 GUID: 0x00066a02300001e0
            ID String: Chassis 0x00066A005000010C, Slot 2, IOC 2
            IO Class: 2000 SubClass: 66a Protocol: 0 Protocol Ver: 1
            VendorID: 0x66a DeviceID: 0x30 Rev: 0x1
            Subsystem: VendorID: 0x66a DeviceID: 0x30
            Capability: 0x33: ST SF WT WF
            Send Depth: 2 Size: 256; RDMA Read Depth: 0 RDMA Size: 4294967295
        2 Services:
            Name: InfiniNIC.InfiniConSys.Control:02
            Id: 0x1000066a00000002
            Name: InfiniNIC.InfiniConSys.Data:02
            Id: 0x1000066a00000102
1 Matching IOUs Found
8.5.1.10.5. Identify connections and links composing the fabric
[goblin1 root@goblin1]# opareport -o links
Getting All Node Records...                                                     
Done Getting All Node Records                                                   
Done Getting All Link Records                                                   
Done Getting All Cable Info Records                                             
Done Getting All SM Info Records                                                
Link Summary

96 Links in Fabric:
Rate NodeGUID          Port Type Name
100g 0x001175010157401b   1 FI   goblin8 hfi1_0
<->  0x00117501025131cb   7 SW   edge2
100g 0x001175010157403d   1 FI   goblin2 hfi1_0
<->  0x00117501025131cb  40 SW   edge2
100g 0x0011750101574053   1 FI   goblin12 hfi1_0
<->  0x00117501025131cb  23 SW   edge2
100g 0x001175010157405c   1 FI   goblin16 hfi1_0
<->  0x00117501025019ab  33 SW   edge1
100g 0x001175010157406c   1 FI   goblin13 hfi1_0
<->  0x00117501025019ab  42 SW   edge1
100g 0x0011750101574071   1 FI   goblin18 hfi1_0
<->  0x00117501025131cb   4 SW   edge2
100g 0x0011750101574074   1 FI   goblin15 hfi1_0
<->  0x00117501025019ab  41 SW   edge1
100g 0x0011750101574077   1 FI   goblin20 hfi1_0
<->  0x00117501025131cb   3 SW   edge2
100g 0x001175010157409d   1 FI   goblin1 hfi1_0
<->  0x00117501025131cb  44 SW   edge2
100g 0x00117501015740bb   1 FI   goblin22 hfi1_0
<->  0x00117501025019ab   2 SW   edge1
100g 0x00117501015740bd   1 FI   goblin3 hfi1_0
<->  0x00117501025131cb  43 SW   edge2
100g 0x00117501015740db   1 FI   goblin21 hfi1_0
<->  0x00117501025019ab  10 SW   edge1
100g 0x00117501015740e0   1 FI   goblin41 hfi1_0
<->  0x00117501025019ab  22 SW   edge1
100g 0x00117501015740e3   1 FI   goblin33 hfi1_0
<->  0x00117501025131cb  48 SW   edge2
100g 0x0011750101574e8b   1 FI   goblin25 hfi1_0
<->  0x00117501025019ab  46 SW   edge1
100g 0x0011750101574f08   1 FI   goblin45 hfi1_0
<->  0x00117501025019ab  18 SW   edge1
100g 0x0011750101574f6c   1 FI   goblin42 hfi1_0
<->  0x00117501025019ab  30 SW   edge1
100g 0x0011750101574fea   1 FI   goblin29 hfi1_0
<->  0x00117501025131cb  32 SW   edge2
100g 0x0011750101575021   1 FI   goblin46 hfi1_0
<->  0x00117501025019ab  25 SW   edge1
100g 0x001175010157504e   1 FI   goblin47 hfi1_0
<->  0x00117501025019ab  17 SW   edge1
100g 0x0011750101575068   1 FI   goblin10 hfi1_0
<->  0x00117501025131cb  24 SW   edge2
100g 0x0011750101575082   1 FI   goblin23 hfi1_0
<->  0x00117501025019ab   9 SW   edge1
100g 0x00117501015750a1   1 FI   goblin48 hfi1_0
<->  0x00117501025019ab  26 SW   edge1
100g 0x001175010157513c   1 FI   goblin34 hfi1_0
<->  0x00117501025131cb  47 SW   edge2
100g 0x0011750101575153   1 FI   goblin35 hfi1_0
<->  0x00117501025131cb  36 SW   edge2
100g 0x001175010157515e   1 FI   goblin4 hfi1_0
<->  0x00117501025131cb  39 SW   edge2
100g 0x0011750101575188   1 FI   goblin17 hfi1_0
<->  0x00117501025131cb  16 SW   edge2
100g 0x00117501015751b8   1 FI   goblin11 hfi1_0
<->  0x00117501025131cb  27 SW   edge2
100g 0x00117501015751c9   1 FI   goblin30 hfi1_0
<->  0x00117501025131cb  19 SW   edge2
100g 0x00117501015751d6   1 FI   goblin6 hfi1_0
<->  0x00117501025131cb   8 SW   edge2
100g 0x00117501015751dd   1 FI   goblin37 hfi1_0
<->  0x00117501025019ab  14 SW   edge1
100g 0x00117501015751df   1 FI   goblin43 hfi1_0
<->  0x00117501025019ab  29 SW   edge1
100g 0x00117501015751e5   1 FI   goblin31 hfi1_0
<->  0x00117501025131cb  20 SW   edge2
100g 0x00117501015751ef   1 FI   goblin38 hfi1_0
<->  0x00117501025019ab   5 SW   edge1
100g 0x00117501015751ff   1 FI   goblin19 hfi1_0
<->  0x00117501025131cb  15 SW   edge2
100g 0x0011750101575f28   1 FI   goblin39 hfi1_0
<->  0x00117501025019ab   6 SW   edge1
100g 0x0011750101575f63   1 FI   goblin26 hfi1_0
<->  0x00117501025019ab  45 SW   edge1
100g 0x0011750101575f6a   1 FI   goblin44 hfi1_0
<->  0x00117501025019ab  21 SW   edge1
100g 0x0011750101575fa1   1 FI   goblin40 hfi1_0
<->  0x00117501025019ab  13 SW   edge1
100g 0x0011750101575fba   1 FI   goblin7 hfi1_0
<->  0x00117501025131cb  11 SW   edge2
100g 0x0011750101575feb   1 FI   goblin14 hfi1_0
<->  0x00117501025019ab  34 SW   edge1
100g 0x001175010157e3d1   1 FI   goblin36 hfi1_0
<->  0x00117501025131cb  35 SW   edge2
100g 0x001175010157e3f0   1 FI   goblin24 hfi1_0
<->  0x00117501025019ab   1 SW   edge1
100g 0x001175010157e3f3   1 FI   goblin32 hfi1_0
<->  0x00117501025131cb  31 SW   edge2
100g 0x001175010157e406   1 FI   goblin27 hfi1_0
<->  0x00117501025019ab  38 SW   edge1
100g 0x001175010157e40e   1 FI   goblin9 hfi1_0
<->  0x00117501025131cb  28 SW   edge2
100g 0x001175010157e418   1 FI   goblin28 hfi1_0
<->  0x00117501025019ab  37 SW   edge1
100g 0x001175010157e427   1 FI   goblin5 hfi1_0
<->  0x00117501025131cb  12 SW   edge2
100g 0x00117501025019ab   3 SW   edge1
<->  0x0011750102513145   3 SW   edge4
100g 0x00117501025019ab   4 SW   edge1
<->  0x0011750102513145   4 SW   edge4
100g 0x00117501025019ab   7 SW   edge1
<->  0x0011750102513145   7 SW   edge4
100g 0x00117501025019ab   8 SW   edge1
<->  0x0011750102513145   8 SW   edge4
100g 0x00117501025019ab  11 SW   edge1
<->  0x0011750102513139  11 SW   edge3
100g 0x00117501025019ab  12 SW   edge1
<->  0x0011750102513139  12 SW   edge3
100g 0x00117501025019ab  15 SW   edge1
<->  0x0011750102513139  15 SW   edge3
100g 0x00117501025019ab  16 SW   edge1
<->  0x0011750102513139  16 SW   edge3
100g 0x00117501025019ab  19 SW   edge1
<->  0x0011750102513145  19 SW   edge4
100g 0x00117501025019ab  20 SW   edge1
<->  0x0011750102513145  20 SW   edge4
100g 0x00117501025019ab  23 SW   edge1
<->  0x0011750102513145  23 SW   edge4
100g 0x00117501025019ab  24 SW   edge1
<->  0x0011750102513145  24 SW   edge4
100g 0x00117501025019ab  27 SW   edge1
<->  0x0011750102513139  27 SW   edge3
100g 0x00117501025019ab  28 SW   edge1
<->  0x0011750102513139  28 SW   edge3
100g 0x00117501025019ab  31 SW   edge1
<->  0x0011750102513139  31 SW   edge3
100g 0x00117501025019ab  32 SW   edge1
<->  0x0011750102513139  32 SW   edge3
100g 0x00117501025019ab  35 SW   edge1
<->  0x0011750102513145  35 SW   edge4
100g 0x00117501025019ab  36 SW   edge1
<->  0x0011750102513145  36 SW   edge4
100g 0x00117501025019ab  39 SW   edge1
<->  0x0011750102513145  39 SW   edge4
100g 0x00117501025019ab  40 SW   edge1
<->  0x0011750102513145  40 SW   edge4
100g 0x00117501025019ab  43 SW   edge1
<->  0x0011750102513139  43 SW   edge3
100g 0x00117501025019ab  44 SW   edge1
<->  0x0011750102513139  44 SW   edge3
100g 0x00117501025019ab  47 SW   edge1
<->  0x0011750102513139  47 SW   edge3
100g 0x00117501025019ab  48 SW   edge1
<->  0x0011750102513139  48 SW   edge3
100g 0x0011750102513139   1 SW   edge3
<->  0x00117501025131cb   1 SW   edge2
100g 0x0011750102513139   2 SW   edge3
<->  0x00117501025131cb   2 SW   edge2
100g 0x0011750102513139   5 SW   edge3
<->  0x00117501025131cb   5 SW   edge2
100g 0x0011750102513139   6 SW   edge3
<->  0x00117501025131cb   6 SW   edge2
100g 0x0011750102513139  17 SW   edge3
<->  0x00117501025131cb  17 SW   edge2
100g 0x0011750102513139  18 SW   edge3
<->  0x00117501025131cb  18 SW   edge2
100g 0x0011750102513139  21 SW   edge3
<->  0x00117501025131cb  21 SW   edge2
100g 0x0011750102513139  22 SW   edge3
<->  0x00117501025131cb  22 SW   edge2
100g 0x0011750102513139  33 SW   edge3
<->  0x00117501025131cb  33 SW   edge2
100g 0x0011750102513139  34 SW   edge3
<->  0x00117501025131cb  34 SW   edge2
100g 0x0011750102513139  37 SW   edge3
<->  0x00117501025131cb  37 SW   edge2
100g 0x0011750102513139  38 SW   edge3
<->  0x00117501025131cb  38 SW   edge2
100g 0x0011750102513145   9 SW   edge4
<->  0x00117501025131cb   9 SW   edge2
100g 0x0011750102513145  10 SW   edge4
<->  0x00117501025131cb  10 SW   edge2
100g 0x0011750102513145  13 SW   edge4
<->  0x00117501025131cb  13 SW   edge2
100g 0x0011750102513145  14 SW   edge4
<->  0x00117501025131cb  14 SW   edge2
100g 0x0011750102513145  25 SW   edge4
<->  0x00117501025131cb  25 SW   edge2
100g 0x0011750102513145  26 SW   edge4
<->  0x00117501025131cb  26 SW   edge2
100g 0x0011750102513145  29 SW   edge4
<->  0x00117501025131cb  29 SW   edge2
100g 0x0011750102513145  30 SW   edge4
<->  0x00117501025131cb  30 SW   edge2
100g 0x0011750102513145  41 SW   edge4
<->  0x00117501025131cb  41 SW   edge2
100g 0x0011750102513145  42 SW   edge4
<->  0x00117501025131cb  42 SW   edge2
100g 0x0011750102513145  45 SW   edge4
<->  0x00117501025131cb  45 SW   edge2
100g 0x0011750102513145  46 SW   edge4
<->  0x00117501025131cb  46 SW   edge2
8.5.1.10.6. Reverse lookup

The following example translates a LID or GUID into the information about the node or port represented.

[root@duster duster]# opareport -o nodes -F lid:5
Getting All Node Records...                                                     
Done Getting All Node Records                                                   
Done Getting All Link Records                                                   
Done Getting All Cable Info Records                                             
Done Getting All SM Info Records                                                
Node Type Summary
Focused on:
    Port:   1 0x0011750101574071
        in Node: 0x0011750101574071 FI goblin2 hfi1_0

48 Connected FIs in Fabric:
    Name: goblin2 hfi1_0
        NodeGUID: 0x0011750101574071 Type: FI
        Ports: 1 PartitionCap: 16 SystemImageGuid: 0x0011750101574071
        BaseVer: 128 SmaVer: 128 VendorID: 0x1175 DeviceID: 0x24f0 Rev: 0x0
    1 Connected Ports:
        PortNum:   1 LID: 0x0005 GUID: 0x0011750101574071
            Neighbor:  Name: edge1
                NodeGUID: 0x00117501025131cb Type: SW PortNum:   4
            Width:    4 Speed:    25Gb Downgraded?  No
1 Matching FIs Found

4 Connected Switches in Fabric:
0 Matching Switches Found

1 Connected SMs in Fabric:
0 Matching SMs Found
----------------------------------------
8.5.1.10.7. Forward lookup

The following example returns information about nodes or IOCs listed by name.

[root@duster root]# opareport -o nodes -F "node:goblin2 hfi1_0"
Getting All Node Records...                                                     
Done Getting All Node Records                                                   
Done Getting All Link Records                                                   
Done Getting All Cable Info Records                                             
Done Getting All SM Info Records                                                
Node Type Summary
Focused on:
    Node: 0x0011750101574071 FI goblin2 hfi1_0

48 Connected FIs in Fabric:
    Name: goblin2 hfi1_0
        NodeGUID: 0x0011750101574071 Type: FI
        Ports: 1 PartitionCap: 16 SystemImageGuid: 0x0011750101574071
        BaseVer: 128 SmaVer: 128 VendorID: 0x1175 DeviceID: 0x24f0 Rev: 0x0
    1 Connected Ports:
        PortNum:   1 LID: 0x0005 GUID: 0x0011750101574071
            Neighbor:  Name: edge1
                NodeGUID: 0x00117501025131cb Type: SW PortNum:   4
            Width:    4 Speed:    25Gb Downgraded?  No
1 Matching FIs Found

4 Connected Switches in Fabric:
0 Matching Switches Found

1 Connected SMs in Fabric:
0 Matching SMs Found
-------------------------------------------------------------------------------
8.5.1.10.8. Generate report for comparison

The following example generates a report so topology verification can be performed against a known good configuration.

Note

To shorten the length of the output, the following example focuses on only one node.

[root@phwtpriv27 ~]$ opareport -o nodes -F node:"goblin02 hfi1_0" -d 5 -P
Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Done Getting vFabric Records
Node Type Summary
Focused on:
    Node: 0x0002b3010100002b FI goblin02 hfi1_0

48 Connected FIs in Fabric:
    Name: goblin02 hfi1_0
        NodeGUID: 0x0002b3010100002b Type: FI
        Ports: 1 PartitionCap: 16 SystemImageGuid: 0x0002b3010100002b
        BaseVer: 128 SmaVer: 128 VendorID: 0x2b3 DeviceID: 0x7220 Rev: 0x60002
    1 Connected Ports:
        PortNum:   1 LID: xxxxxx GUID: 0x0002b3010100002b
            Neighbor:  Name: MOOSE_STL_SWITCH0
                NodeGUID: 0x0002b30102000000 Type: SW PortNum:   5
            LocalPort: 1   PortState: Active                 PhysState: LinkUp
            IsSMConfigurationStarted: True  NeighborNormal: True
            PortType: Standard
            LID:    xxxxxx              LMC: 0               Subnet: 0xfe80000000000000
            SMLID:  xxxxxx   SMSL:  0   RespTimeout:    8 ms  SubnetTimeout:  536 ms
            M_KEY:  0x0000000000000000  Lease:     0 s          Protect: Read-only
            MTU  Supported: (0x6) 8192 bytes
            VLStallCount (per VL):  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            MTU  Active by VL:
            00: 8192 01:    0 02:    0 03:    0 04:    0 05:    0 06:    0 07:    0
            08:    0 09:    0 10:    0 11:    0 12:    0 13:    0 14:    0 15: 2048
            16:    0 17:    0 18:    0 19:    0 20:    0 21:    0 22:    0 23:    0
            24:    0 25:    0 26:    0 27:    0 28:    0 29:    0 30:    0 31:    0
            LinkWidth: Active:       4  Supported:    1,2,3,4  Enabled:    1,2,3,4
            LinkWidthDnGrade: ActiveTx:  4 Rx:  4 Supported:    1,2,3,4  Enabled:        3,4
            PortLinkMode: Active: STL      Supported: STL              Enabled: STL
            LinkSpeed: Active:    25Gb  Supported: 25Gb  Enabled: 25Gb
            SM_TrapQP: 0x0  SA_QP: 0x1 IPAddr Prim/Sec: :: / 0.0.0.0
            VLs:       Active:     8+1  Supported:        8+1
                HOQLife (Per VL):
                VL 0: 0x0 VL 1: 0x0 VL 2: 0x0 VL 3: 0x0 VL 4: 0x0
                VL 5: 0x0 VL 6: 0x0 VL 7: 0x0 VL 8: 0x0 VL 9: 0x0
                VL10: 0x0 VL11: 0x0 VL12: 0x0 VL13: 0x0 VL14: 0x0
                VL15: 0x0 VL16: 0x0 VL17: 0x0 VL18: 0x0 VL19: 0x0
                VL20: 0x0 VL21: 0x0 VL22: 0x0 VL23: 0x0 VL24: 0x0
                VL25: 0x0 VL26: 0x0 VL27: 0x0 VL28: 0x0 VL29: 0x0
                VL30: 0x0 VL31: 0x0

            VL Arb Cap:  High:      16        Low:         16  HiLimit:    0 PreemptLimit    0
            VLFlowControlDisabledMask: 0x00000000
            NeighborMode  MgmtAllowed:  No  FwAuthenBypass: Off  NeighborNodeType: Switch
            Capability 0x00000000: -
            Capability3 0x0008: SS
            Violations: M_Key: xxxxx P_Key: xxxxx Q_Key: xxxxx
            PortMode ActiveOptimize: Off PassThrough: Off VLMarker: Off
            FlitCtrlInterleave Distance Max:  0 Enabled:  0
            MaxNestLevelTxEnabled: 10 MaxNestLevelRxSupported: 10
            SmallPktLimit: 0x07 MaxSmallPktLimit: 0xff PreemptionLimit: 0x10
            FlitCtrlPreemption MinInital: 80 MinTail: 80 LargePktLim: 0x07
            BufferUnits: VL15Init 0x0040; VL15CreditRate 0x00; CreditAck 0x0; BufferAlloc 0x3
            PortErrorActions: 0x172000: CE-UVLMCE-BCDCE-BTDCE-BHDR-BVLM
            ReplayDepth Buffer 0x00; Wire xxxx
            DiagCode: 0x0000
            OverallBufferSpace: 0x0900
            P_Key Enforcement: In: Off Out: Off
            QSFP: ActiveCu , 1m   STL Simulator
                  Power Class 5, 4.0W max S/N 0x0000000000033
                  OUI 0x0002B3
                  Cable Type: Linear active copper cable
                  Speed Sup: 0 Gb
1 Matching FIs Found

1 Connected Switches in Fabric:
0 Matching Switches Found

1 Connected SMs in Fabric:
0 Matching SMs Found
----------------------------------------------------------------------------

8.5.1.11. Snapshots

You can take a snapshot of the fabric state for later offline analysis using the -o snapshot report. This report generates an XML snapshot of the present fabric status in a format that opareport can parse.

Note

Cornelis recommends that you do not develop your own tools against this format because it may change in future versions of opareport.

The snapshot capability can be used to provide powerful analysis capabilities. Multiple reports can be run against the exact same fabric snapshot, which saves time by not requiring the subsequent reports to query the fabric. Also, historic snapshots can be retained for later offline analysis or historical tracking of the fabric.

When a snapshot is generated, no additional -o options are allowed during the run and certain opareport options are ignored. These include: -F, -P, -H, and -N. However, the following options are valid:

  • -s includes port counters in the snapshot.

  • -r includes switch routing tables in the snapshot.

  • -V includes QoS VL-related tables in the snapshot.

  • -i, -L, -a, and -C control the port counters.

Note

Quarantined nodes cannot be obtained from a snapshot.

  • opareport -o quarantinednodes -X snapshot does not give the quarantined nodes on a snapshot of the same fabric.

  • Use opareport -o quarantinednodes to return the quarantined nodes on a fabric with quarantined nodes.

After a snapshot has been generated, it may then be used as input to generate many types of opareport reports. To do this, use the -X snapshot_input option, where the snapshot_input file is the output from a previous snapshot run. When using a snapshot as input, the fabric is not accessed and the node running opareport does not need to be attached to the fabric. Because this is a static report, certain options are not available, including -i, -a, -C, -h HFI, and -p port.

The report generated from the snapshot includes port counters only if the original snapshot was run with the -s option. If not, reports such as -o errors are not permitted against the snapshot.

Similarly, certain reports are permitted only if the original snapshot was run with the -r option. This includes: -o linear, -o mcast,-o portusage, -o pathusage, -o treepathusage, and -o route.

If you want to use standard input (stdin) for the snapshot file, then specify -X. This can be helpful if snapshots are piped through gzip/gunzip to conserve disk space.

Note

Limitations of -o route:

  • The Path Records reported may not be complete. The report shows the minimum valid value or an invalid value because certain fields such as SLID, SL, PKey, MTU, Rate, and PktLifeTime are not available. These values do not impact the actual route shown.

  • Some routes reported may not be incomplete or not available to applications.

Inversion of Control - This is the custom-written portions of a computer program that receive the flow of control from a generic framework.

Globally Unique Identifier - This is a 128-bit text string generated when a unique reference number is needed to identify a component (for example, hardware, software, node) on a computer or network.

Local Identifier - This is a 24-bit address local to a subnet that is assigned to end nodes by the subnet manager, providing additional features related to routing algorithms.

Double Data Rate - This transfers data on both the rising and falling edges of the clock signal.

Single Data Rate - This is a computer network communications link used in computing; approximately 2.5 Gbps.

Performance Management Agent

Inter-Switch Link - This is a Cisco Systems-proprietary protocol that maintains VLAN information in Ethernet frames as traffic flows between switches and routers, or switches and switches.

LID Mask Control - This method provides the assignment multiple local identifiers (LIDs) to a single physical port.

Subnet Manager or Subnet Management

Subnet Management Agent

Maximum Transmission Unit - This is the size of the largest protocol data unit or data packet that can be sent in a single network layer transaction.

Fabric Manager - This is an OPX Software component responsible for managing the fabric using management packets over a dedicated virtual lane (VL15).

Group Identifier - This is a unique identifier assigned to a group of users on a system, allowing them to access files and directories with common permissions.

Message Passing Interface - This is a standardized means of exchanging messages between multiple computers running a parallel program across distributed memory

Virtual Lane - This feature allows multiple, multiplexed, independent data to stream onto a single physical link.

Source Local Identifier

Service Level

Partition Key - This is a unique ID assigned to an InfiniBand or vFabric partition.