7.3.2.5. Update the OPX Software

Cornelis Technical Documentation

7.3.2.5. Update the OPX Software

This section provides information and procedures to update to the OPX Software.

7.3.2.5.1. Prerequisites

Prior to updating the OPX Software, ensure the following items have been completed:

Review the Release Notes for a list of compatible software.
Uninstall all versions of third-party IB stacks.
Back up the following configuration files, if applicable, in case the upgrade fails:
- /etc/opa-fm/opafm.xml
- /etc/opa/*
- /etc/sysconfig/opa/*
- /var/usr/lib/opa/analysis/baseline/*
Note
Refer to the OS documentation for a list of any other OS-specific files that should be included in any backups.

7.3.2.5.2. Update Sequence

Optimally, to update the Omni-Path Software, you perform the following sequence:

Stop all standby Fabric Managers.
Stop the primary Fabric Manager.
Upgrade the nodes.
Reboot the nodes (if autostart was not enabled).
Update the opafm.xml file, as needed.
Start the Fabric Manager.

7.3.2.5.1. Stopping the Fabric Manager

To stop the Fabric Manager, perform the following steps:

Log into the Fabric Manager system as root or as a user with root privileges.
Stop the Fabric Manager:
```
systemctl stop opafm
```

7.3.2.5.2. Upgrading the OPX Software

To upgrade the CN5000 OPX Software, you first upgrade the management node then the remaining servers.

7.3.2.5.2.1. Assumption

You have downloaded the required software RPMs for upgrade.
You have stopped all standby FMs, followed by the primary FM.
You are logged in to the target Management Node.

7.3.2.5.2.2. Procedure

The following instructions provide the steps for upgrading the host software on the first management node and then the remaining nodes.

Note

If you plan to use a local repository and the TGZ package from the Cornelis Customer Center, download and extract the package to the management node.

Download and install the OPX Software package:

Create a repository folder or confirm that the following already exists.
- For RHEL:
```
/etc/yum.repos.d/
```
- For SLES:
```
/etc/zypp/repos.d/
```
- For Ubuntu:
```
/etc/apt/sources.list.d/
```

Create a repository file.

For RHEL:

sudo vi /etc/yum.repos.d/cornelis-repository.repo

For SLES:

sudo vi /etc/zypp/repos.d/cornelis-repository.repo

For Ubuntu:

sudo vi /etc/apt/sources.list.d/cornelis-repository.list

Edit the repository file to add the following content.

Using a local repository and RHEL or SLES TGZ package:

[Cornelis-Package]
name=Cornelis Repository
baseurl=file:///<file path to local downloaded updated tar file>
autorefresh=1  (SLES ONLY)
enable=1
gpgcheck=1

For Ubuntu, perform the following

Import the GPG public key.

sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/cornelis.gpg <file path to local downloaded updated tar file>/CN5000-Packages-Public-GPG-Key.asc

Add content to the repository (.list) file.

# cat /etc/apt/sources.list.d/cornelis-repository.list
deb [trusted=yes] file:<file path to local downloaded updated tar file>/ ./

Clean and update the package Index.

For RHEL:
```
sudo dnf clean all
sudo dnf makecache
```
For SLES:
```
sudo zypper clean
sudo zypper refresh
```
For Ubuntu:
```
sudo apt clean
sudo apt update
```

Determine the meta package you need to use for installation.
Note
Ubuntu uses dashes in place of underscores in the following list.
- cn5000_pkgs_non_gpu_meta - CPU-based installation
- cn5000_pkgs_cuda_meta - NVIDIA GPU installation
- cn5000_pkgs_rocm_meta - AMD GPU installation
Caution
Installing meta packages will override existing packages. If multiple versions are desired (for example, GPU/non-GPU), additional components can be installed directly using the package manager, but not using the meta packages.
Upgrade the package using one of the following commands.
- For RHEL:
```
sudo dnf install <META_PACKAGE_NAME>
```
- For SLES:
```
sudo zypper install <META_PACKAGE_NAME>
```
  Note
  If zypper prompts for confirmation, you can select the default to continue.
- For Ubuntu:
```
sudo apt install <META_PACKAGE_NAME>
```
  Note
  If you receive an error related to required dependencies that are not the latest versions available on Ubuntu, Cornelis recommends that you install and use aptitude in place of apt in the command.
  Alternately, you can use apt or apt-get with --mark-auto to manually install the dependencies.
Perform the following:
- For RHEL and SLES, continue to the next step to download, build, and install the kmod-opxs-kernel-updates packages.
- For Ubuntu, reboot the server to finalize the installation, then go to Step 13 to install the software to each remaining server.
  Note
  The opxs-modules-dkms packages are already installed with the Ubuntu meta package.

Download, build, and install the kmod-opxs-kernel-updates packages (RHEL and SLES only):

Copy the relevant RPM source file (src.rpm) to a local directory.
Important
- Ensure you are using the correct src.rpm file to build and install the version of the opx-kernel-updates relevant to your hardware configuration (AMD GPU, NVIDIA GPU, or CPU-only). Three opxs-kernel-updates RPM source files are provided, one each for AMD GPU, NVIDIA GPU, and CPU-only.
- RHEL Only: To rebuild the source RPM, you must install kernel-rpm-macros (for AMD GPUs) and kernel-abi-stablelists.
- SLES Only: You need to build unifdef before running rpmbuild:
```
cd /lib/modules/$(uname -r)/source/scripts
gcc -o unifdef unifdef.c
```

Build the source RPM.

rpmbuild --rebuild --define "_topdir <local_directory>" --define 'dist %{nil}' --target 
x86_64 --define 'kver $(uname -r)' <src.rpm_file>

Verify the RPMs are built by changing directories to the <local_directory>/RPMS/x86_64 directory and running ls -1.
Output should be:
```
kmod-opxs-kernel-updates-<build_version>.x86_64.rpm
opxs-kernel-updates-devel-<build_version>.x86_64.rpm
```

Upgrade the new RPMs.

For RHEL:

sudo dnf install kmod-opxs-kernel-updates-<build_version>.x86_64.rpm 
opxs-kernel-updates-devel-<build_version>.x86_64.rpm

For SLES:

zypper --no-gpg-checks install opxs-kernel-updates-devel-<build_version>.x86_64.rpm 
opxs-kernel-updates-kmp-default-<build_version>-50.x86_64.rpm

Reboot the primary node to apply changes.

Install the OPX Software to each remaining server:

Copy the Repository folder, the tar file (if using a local repository), and the <local_directory> containing the newly built RPMs, to each remaining server.
Important
The folders and files must be in the same directories as was in the management node.
Extract the TGZ package.
Upgrade the software on the remaining nodes using the commands in steps 6 and 11.
Reboot the rest of the nodes to apply changes.
After update, disable the repository on every node using one of the following methods:
- For RHEL, run the following commands:
```
sudo dnf config-manager --set-disabled <repo>
```
- For RHEL and SLES, in the repository file, change enable to 0.
- For Ubuntu, in the repository (.list) file, comment out the following line:
```
#deb [trusted=yes] file:/<file path to local downloaded updated tar file>/ ./
```

7.3.2.5.3. Update the Fabric Manager opafm Configuration File

The opafm configuration file is designed so that any new parameters that might not appear in your old opafm.xml file will automatically be set to their defaults in the software. If you have customized your pre-upgrade opafm.xml file, it is recommended that you manually update your new opafm.xml file with the latest comments and defaults. This will aid you with future customizations and make comparisons with the opafm.xml in future upgrades easier.

Important

The opafm configuration file must be the same on all management nodes.

7.3.2.5.3.1. Assumption

You have upgraded the Fabric Manager, accepting all the defaults.

7.3.2.5.3.2. Procedures

Perform the following steps to transfer customizations into a new opafm.xml file:

Create a new opafm.xml using the /usr/share/opa-fm/opafm.xml file as a base for the new file by copying this read-only file to a temporary location to make the subsequent edits.
Update the new opafm.xml with the required customizations.
Use the new file from the previous step to replace the /etc/opa-fm/opafm.xml file.
Restart the Fabric Manager.
- If you previously upgraded the management node, type reboot and press Enter.
  Note
  If the Omni-Path Autostart was disabled, restart the primary Fabric Manager, followed by the standby Fabric Manager. Refer to Starting the Fabric Manager.
- If you only updated the opafm.xml file, type systemctl restart opafm and press Enter.
Copy the opafm.xml file to the standby Fabric Managers, replacing the /etc/opa-fm/opafm.xml file.
Restart the standby Fabric Managers.

7.3.2.5.4. Starting the Fabric Manager

This section provides instructions for starting the Fabric Manager.

Important

The Fabric Manager is always run from a node connected to the latest generation switch (for example, a CN5000 switch).

7.3.2.5.4.1. Prerequisites

Prior to starting the Fabric Manager, you must edit the opafm.xml file (located under /etc/opa-fm/) and ensure the following:

HFI 1, port 2 is enabled for fm0:
```
<Start>1</Start>  <-- 1 signifies enabled
<Name>fm0</Name> 
<Hfi>1</Hfi> 
<Port>2</Port>
```
Note
Also note that a single port SuperNIC appears as dual port in opainfo output.
CableInfoPolicy is set to None:
```
<CableInfoPolicy>None</CableInfoPolicy>
```

7.3.2.5.4.2. Procedure

To start the Fabric Manager, perform the following steps:

Log into the Fabric Manager system as root or as a user with root privileges.
Start the Fabric Manager:
```
systemctl start opafm
```
Verify that all the tasks are up and running:
```
systemctl status opafm
```
Note
The default configuration runs Fabric Manager on port 2 of the first SuperNIC in the system.

Would you like to provide feedback? Just click here to suggest edits.

Cornelis Technical Documentation

7.3.2.5. Update the OPX Software

7.3.2.5.1. Prerequisites

Note

7.3.2.5.2. Update Sequence

7.3.2.5.1. Stopping the Fabric Manager

7.3.2.5.2. Upgrading the OPX Software

7.3.2.5.2.1. Assumption

7.3.2.5.2.2. Procedure

Note

Note

Caution

Note

Note

Note

Important

Important

7.3.2.5.3. Update the Fabric Manager opafm Configuration File

Important

7.3.2.5.3.1. Assumption

7.3.2.5.3.2. Procedures

Note

7.3.2.5.4. Starting the Fabric Manager

Important

7.3.2.5.4.1. Prerequisites

Note

7.3.2.5.4.2. Procedure

Note

Search results