When an application creates or joins a multicast group, the SM determines whether there is a path with sufficient speed and MTU to meet the multicast group's requested capabilities.

To address this situation, the SM performs stricter multicast checking at the Join/Create time. This means a multicast Join/Create is rejected if any switch-to-switch links do not have at least the requested MTU or rate for the multicast group, reducing the chance that a simple fabric failure or change (for example, the loss of one link) could make the group unusable.

The DisableStrictCheck parameter controls this capability. When set to 1, this parameter disables the strict checking and accepts Join/Create requests for which at least one viable fabric path exists. By default, the parameter is set to 0, which allows for more strict checking.

In addition, the MLIDTableCap parameter configures the maximum number of Multicast LIDs available in the fabric. This must be set to a value less than or equal to the smallest Multicast Forwarding Table (MFT) size of all the Switches that may be in the fabric. It defaults to 1024, which is below the capability of all current CN5000 Omni-Path Switches. Using a value below the Switches' capability can prevent errant applications from creating an excessive number of multicast groups.

IPv6—and potentially other applications—can create a large number of multicast groups. In IPv6, each SuperNIC/TFI port requires its own Solicited-Node multicast group. As a result, the number of multicast groups can grow rapidly and may exceed the available MLIDTableCap capacity. For example, a fabric containing 10,000 nodes running IPv6 would require more than 10,000 multicast groups.

To scale beyond these limits, the SM can assign a single MLID to multiple multicast groups. When MLIDs are shared, the associated routes and destination lists are also shared. This may cause some end nodes to receive multicast packets they did not request. However, these unwanted packets are automatically discarded by the SuperNIC/TFI transport layer and do not affect application behavior.

The SM supports MLID sharing through MLIDShare sections in the configuration file. These sections define groups of multicast addresses that share a common pool of Multicast LIDs. By reducing the number of MLIDs required, MLID sharing helps conserve hardware MLID table resources and leaves more capacity available for other multicast uses. By default, the SM allocates a pool of 500 MLIDs that is shared across all IPv6 Solicited-Node multicast groups:

You can also configure the maximum number of MLIDs that a single PKey can consume within an MLIDShare pool. In environments with many vFabrics, this prevents any one vFabric from monopolizing the available MLIDs and exhausting the shared pool.

The first end node that joins a multicast group also creates the multicast group. When a multicast group is created, critical parameters such as the MTU and speed of the multicast group are also established. The selection of these values must carefully balance the performance of the multicast group against the capabilities of the hardware, which may need to participate in the group in the future. For example, if an application on a SuperNIC with a 4K MTU creates a 4K multicast group, it prevents subsequent joins of the group by 2K MTU SuperNICs.

Some ULPs and applications, such as IPoIB, require key multicast groups, such as the IPv4 broadcast group, to be pre-created by the SM.

Pre-created multicast group configurations are specified in the MulticastGroup sections of the SM configuration files. When the multicast groups are pre-created, their MTU and speed are defined by the SM configuration file, allowing you to be able to account for anticipated hardware capabilities and required performance.

To simplify typical configurations, a MulticastGroup section, which does not explicitly specify any MGIDs, will implicitly include all the multicast groups specified by the IPv4 and IPv6 standards for IPoIB. As such, typical clusters using IPoIB do not need to explicitly list the MGIDs for IPoIB.

Multicast routing is performed by computing a spanning tree for the fabric. When constructing a spanning tree for a multicast group, MTU and rate must be considered. If the SM first constructs a spanning tree for the largest MTU and rate found in the fabric and if that spanning tree is complete (that is, includes all switches) then that spanning tree is sufficient for all groups as it will support all smaller MTUs and rates. If it is not complete, the SM will use MTUs and rates smaller than the maximum until a complete spanning tree is computed. Using a common spanning tree reduces the computational time required in fabric programming. The spanning tree has a root switch and spans throughout the fabric to reach all of the switches and SuperNICs that are members of the multicast group. Since this tree is common across multicast groups, it is optimized to compute the MFTs data associated with each switch only once. This common data is augmented with ports participating in a given multicast group at the switch when programming the MFT.

The FM allows the root of the spanning tree to be configured using the Sm.Multicast.RootSelectionAlgorithm parameter. A goal of the spanning tree calculation is to select a switch at the center of the fabric for the root of the spanning tree. A good option would be a switch that has the least total cost of all other switches. Another option is a switch that has the least worst-case cost to other switches.

Concerning fabric change, the root switch using one of these cost options can change, resulting in a reconstruction of the spanning tree and potential disruption due to reprogramming of the MFTs. The SM’s MinCostImprovement parameter can determine how much improvement is needed before a new spanning tree root is selected. Disruption to in-flight multicast traffic can be avoided or limited to cases where the fabric has changed significantly enough to provide sufficient benefit to justify a change by using these parameters.

The SM’s DB Sync capability synchronizes the multicast root between the primary and standby SMs. During SM failover the multicast root can be retained and limit the disruption to multicast traffic in flight.

A complete tree unconditionally includes all switches. When SuperNICs request to join or leave the multicast group, the SM only needs to program the switch immediately next to the SuperNIC. This is optimized to program only those MFT blocks that have changed since the last sweep.

A pruned tree omits switches that do not have SuperNICs as members of the group, as well as intermediate switches that do not need to be in the group. A pruned tree reduces multicast traffic internal to the fabric when only a small subset of nodes are part of a given multicast group. However, the time to add or remove SuperNICs from the group can be significantly higher as many intermediate switches may need to also be programmed for the group.

The default is a complete tree. This has been found to work very well in HPC environments. Such environments typically have very little multicast traffic with the vast majority of traffic being IPoIB ARP packets that need to be broadcast to all nodes running IPoIB. The default allows IPoIB hosts to come up and down quicker as may be common in environments that restart compute nodes between jobs.