Skip to main content

Cornelis Technical Documentation

7.2.7. Subnet Manager Monitoring

This section provides information on the subnet manager monitoring and debug capabilities.

7.2.7.1. Fabric Sweeping

The SM periodically sweeps the fabric, during which the fabric is analyzed, routes are computed, and the Switches and SuperNICs are configured. The SM sweep algorithms attempt to:

  • Balance responsiveness to fabric changes while limiting fabric overhead of the SM

  • Efficiently analyze and configure the fabric

  • Detect and handle potential hardware limitations

  • Handle the possibility that the fabric is changing while being analyzed or configured

The SM performs sweeps at fixed intervals and also immediately performs a sweep when a switch reports a port state change trap. Such traps indicate a link has come up or down. Generally, traps trigger rapid sweeps to respond to fabric changes. By default, SM sweeps are performed every five minutes. This interval may be changed by the user; however, short sweep intervals may increase fabric overhead.

The SM protocol performs sweep operations over VL15. By default, VL15 operates without flow control, but users can enable flow control if needed. Since the fabric may change while the SM sweeps, and VL15 often lacks flow control, some packets may be lost during the sweep.

To optimize sweep performance, the SM issues multiple concurrent packets to a given device or the fabric. It carefully balances the number of packets sent at once between the hardware's capabilities and the goal of accelerating the sweep.

An administrator may also choose to force a sweep. This is generally not necessary when using fabrics constructed using Cornelis-supplied switches. Sweeps can be forced via /usr/lib/opa-fm/bin/opafmcmdsmForceSweep.

7.2.7.2. SM Logging and Debug

When nodes appear or disappear from the fabric, a message is logged. The SM Logging/Debug section defines how many messages are logged, and if additional SM sweep and SA query debug information is logged. The following is a sample of the SM Logging/Debug section of the Fabric Manager configuration file.

<!-- **************** SM Logging/Debug **************************** -->
<!-- When nodes appear or disappear from the fabric, a message is logged -->
<!-- This can set a threshold on how many such messages to output per -->
<!-- sweep.  Once NodeAppearanceMsgThreshold messages are logged in a -->
<!-- given sweep, the remainder are output at a lower log level (INFO) -->
<!-- Hence avoiding excessive log messages when significant -->
<!-- fabric changes occur.  0 means no limit. -->
<NodeAppearanceMsgThreshold>100</NodeAppearanceMsgThreshold>
<SmPerfDebug>0</SmPerfDebug> <!-- log additional SM sweep info -->
<SaPerfDebug>0</SaPerfDebug> <!-- log additional SA query info -->

The parameters in the following table control SM Logging.

Table 37. SM Logging and Debug Parameters

Parameter

Default Value

Description

NodeAppearanceMsgThreshold

100

This can set a threshold on how many log messages to output per sweep. Once NodeAppearanceMsgThreshold messages are logged in a given sweep, the remainder are output at a lower log level (INFO) therefore avoiding excessive log messages when significant fabric changes occur.

0 means no limit.

SmPerfDebug

0

If 1, then log additional SM sweep info.

SaPerfDebug

0

If 1, then log additional SA query info.

Log additional SA query info SM_0_sa_debug_perf:dec

  • <Debug>0</Debug> – #SM_0_debug:dec

  • <RmppDebug>0</RmppDebug> – #SM_0_sa_debug_rmpp:dec



7.2.7.3. SM Overrides of the Common.Shared Parameters

The Common.Shared parameters can be overridden in the SM using the parameters described in the following table.

Table 38. Additional SM Parameters

Parameter

Default Value

Description

Priority

0

0 to 15, higher wins.

ElevatedPriority

0

0 to 15, higher wins.

LogLevel

2

NOTE: Overrides the Common.Shared LogLevel Settings.

Sets log level option for SM:

  • 0 = disable the vast majority of logging output.

  • 1 = fatal, error, warn (syslog CRIT, ERR, WARN).

  • 2 = +notice, progress (syslog NOTICE, INFO).

  • 3 = +INFO (syslog DEBUG).

  • 4 = +VERBOSE and some packet data (syslog DEBUG).

  • 5 = +debug trace info (syslog DEBUG).

LogFile

NOTE: Overrides Common.Shared settings.

Sets log output location for SM. By default (or if this parameter is empty) log output is accomplished using syslog. However, if a LogFile is specified, logging will be done to the given file. LogMode further controls logging.

SyslogFacility

Local6

NOTE: Overrides Common.Shared settings.

For the Host Fabric Manager, controls what syslog facility code is used for log messages. Allowed values are: auth, authpriv, cron, daemon, ftp, kern, local0-local7, lpr, mail, news, syslog, user, or uucp.

ConfigConsistencyCheckLevel

2

Controls the Configuration Consistency Check for SM. If specified for an individual instance of SM, overrides Shared settings. Checking can be completely disabled or can be set to take action by deactivating Standby SM if the configuration does not pass the consistency check criteria.

  • 0 = disable Configuration Consistency Checking

  • 1 = enable Configuration Consistency Checking without taking action (only log a message)

  • 2= enable Configuration Consistency Checking and take action (log message and move standby to inactive state)



7.2.7.4. LID

LID can be set for this SM as shown in the following table.

Note

If you change the default, and you are using multiple Fabric Managers in a fabric, it is recommended that you use the same SM LID setting on all Fabric Managers in the fabric.

Table 39. Additional SM LID Parameter

Parameter

Default Value

Description

LID

0

LID for this SM, 0=pick any available.

0 is recommended.



Subnet Manager or Subnet Management

Subnet Administrator/Administration

Local Identifier - This is a 24-bit address local to a subnet that is assigned to end nodes by the subnet manager, providing additional features related to routing algorithms.