Skip to main content

Cornelis Technical Documentation

3.1. Introduction

3.1.1. Intended Audience

This document is intended for hardware installers, fabric administrators, and other personnel with similar responsibilities and experience.

3.1.2. Layout and Document Organization

This document provides a sequential approach to deploy your CN5000 Omni-Path Fabric. This guide includes designing space for the Omni-Path hardware, installing the hardware, software, and updating firmware. The guide ends with configuring and then verifying the fabric.

The document follows a sequential deployment model across the fabric lifecycle:

  • Hardware Requirements and Site Preparation: Designing space, cooling, power, liquid cooling requirements, and cabling topology.

  • Hardware Installation: Installing and cabling SuperNIC adapters and CN6000 switches, followed by initial power-on and port mode configuration.

  • Host Software Installation: Installing the OPX Software stack, configuring prerequisites (IOMMU, GPU/accelerator drivers), and building kernel modules.

  • Fabric Configuration: Starting the Fabric Manager, configuring Boot-over-Fabric/PXE, IPoIB network interfaces, and high-availability bonding.

  • Fabric Verification: Validating server links, running fabric sweeps, testing cable topologies against expected layouts, and verifying routing deadlock-freedom.

3.1.2.1. Installation Overview

The following steps summarize the CN5000 Omni-Path hardware and software installation and fabric configuration process. Each step is covered in detail in the sections that follow.

  1. Review all release notes related to the version(s) of Omni-Path software and firmware being installed.

  2. Install the Omni-Path SuperNICs in servers.

  3. Install the Omni-Path Switches/Omni-Path Director Class Switches in racks.

  4. Connect the cabling.

  5. Power up all fabric hardware.

  6. Verify the firmware versions on the switches and SuperNICs.

    If an update is required, refer to the CN5000 Maintenance and Troubleshooting Guide.

  7. Download the OPX Software from the Cornelis Customer Center.

  8. Install and configure the OPX Software.

  9. Reboot all servers after the software installation and configuration are complete.

  10. Start the primary Fabric Manager.

  11. Verify the fabric is working correctly using Omni-Path verification tools.

  12. Tune the fabric to optimize performance using the CN5000 Performance Tuning Guide.