2.2.2. Software
For software applications, Omni-Path software maintains compatibility with existing CN5000 Omni-Path Fabric and InfiniBand APIs through the open-source OFA software stack.
2.2.2.1. Omni-Path Driver
The CN5000 Omni-Path SuperNIC kernel module (hfi1) provides low-level hardware support. It is the base driver for both MPI programs and general protocols such as IPoIB. The driver also supplies the Subnet Management Agent (SMA) component.
The following are optional configurable components:
IPoIB Network Interface
This component is required for TCP/IP networking for running IP traffic over the CN5000 Omni-Path Fabric link. This component is not enabled by default. See CN5000 Fabric Installation Guide, Configure IPoIB Network Interface for details on enabling this component.
SCSI RDMA Protocol (SRP)
This component does not run until the module is loaded and SRP devices on the fabric have been discovered.
2.2.2.2. OPX Software APIs
APIs provide a set of common interfaces for applications and services to communicate with each other and to access services within the software architecture. The following table overviews the APIs supported by the CN5000 OPX Software stack.
API | Description |
|---|---|
Sockets | Enables applications to communicate with each other using standard interfaces locally or over a distributed network. |
RDMA verbs | Provides access to RDMA capable devices. |
IB Connection Manager | Establishes connections between two verbs queue pairs on one or two nodes. |
RDMA Connection Manager | Provides a sockets-like method for connection establishment. |
OpenFabrics Interface libfabric | Extends the OpenFabrics APIs to increase application support and improve hardware abstraction. Framework features include connection and addressing services, message queues, RDMA transfers, and others; includes support for the existing verbs capabilities. OFI is an open-source project that is actively evolving APIs, sample providers, sample applications, and infrastructure implementation. |
NetDev | In Linux, NetDev interfaces provide access to network devices, their features, and their state. For Omni-Path, NetDev interfaces are created for IP communications over Omni-Path via IPoIB. |
Linux Virtual File System | Provides the abstraction for the traditional system calls of open, close, read, and write. |
SCSI low-level driver | Implements the delivery of SCSI commands to a SCSI device. |
MPI | Standardized API for implementing HPC computational applications. Supported implementations include: Intel OneAPI, Open MPI, MVAPICH2, and others. Additional HPC middleware supported through libfabric includes OpenSHMEM and NCCL/RCCL collective communication libraries. |
Unified Extensible Firmware Interface (UEFI) Specification | Defines the specification for BIOS work, including the multi-core processor (MCP) packaging of the SuperNIC interactions. The specification details booting, including booting over fabric. Omni-Path includes a UEFI BIOS extension to enable PXE boot of nodes over the CN5000 Omni-Path Fabric. |