7.3.2.6. Update the OPX Software

Cornelis Technical Documentation

7.3.2.6. Update the OPX Software

This section provides information and procedures to update to the OPX Software.

7.3.2.6.1. Prerequisites

Prior to updating the OPX Software, ensure the following items have been completed:

Review the Release Notes for a list of compatible software.
Uninstall all versions of third-party IB stacks.
Back up the following configuration files, if applicable, in case the upgrade fails:
- /etc/opa-fm/opafm.xml
- /etc/opa/*
- /etc/sysconfig/opa/*
- /var/usr/lib/opa/analysis/baseline/*
Note
Refer to the OS documentation for a list of any other OS-specific files that should be included in any backups.

7.3.2.6.2. Update Sequence

Optimally, to update the OPX Software, you perform the following sequence:

Stop all standby Fabric Managers.
Stop the primary Fabric Manager.
Upgrade the nodes.
Reboot the nodes (if autostart was not enabled).
Update the opafm.xml file, as needed.
Start the Fabric Manager.

7.3.2.6.1. Stopping the Fabric Manager

To stop the Fabric Manager, perform the following steps:

Log into the Fabric Manager system as root or as a user with root privileges.
Stop the Fabric Manager:
```
systemctl stop opafm
```

7.3.2.6.2. Resolving CUDA Dependency Errors for an OPX Software Upgrade

If CUDA is not installed in your local/system locations, a dependency error may occur. To work around the missing CUDA dependencies during an OPX Software upgrade, you can install a "dummy" RPM package that declares the required libraries as present without actually providing them. On RPM-based systems such as RHEL and SLES, you create this package with the dummy_cuda_provides.spec file.

Once the dummy RPM package is installed, the OPX Software requires only the NVIDIA kernel driver to be present on the system. Note that most older CUDA versions typically work with the latest NVIDIA kernel driver.

Procedures

Create the dummy_cuda_provides.spec file with the following text.

Note

Replace <CUDA version> with the version you are using.

Name:      dummy-cuda-provides
Version:   1.0
Release:   1
Summary:   A dummy package to provide cuda packages
License:   MIT
BuildArch: noarch
 
# List your virtual provides here
Provides:  libcuda.so.1()(64bit)
Provides:  libcudart.so.<CUDA version>()(64bit)
Provides:  libcudart.so.13(libcudart.so.<CUDA version>)(64bit)
Provides:  libNVIDIA-ml.so.1()(64bit)
 
%description
This package does not contain any files. It only provides virtual capabilities
to satisfy dependency checks.

%prep
# No prep needed

%build
# No build needed

%install
# No install needed

%files
# No files to include

Build the dummy-cuda-provides-1.0-1.noarch.rpm.
```
$ rpmbuild -ba dummy_cuda_provides.spec
```
On all nodes, install the created /home/USER/rpmbuild/RPMS/noarch/dummy-cuda-provides-1.0-1.noarch.rpm.
Install the CUDA toolkits on a network-mounted drive. Refer to NVIDIA documentation.
Note
This step is required once per new CUDA version.
Install the latest NVIDIA kernel driver on all nodes. Refer to NVIDIA documentation.
Note
This step is required with every software upgrade.
Upgrade the latest OPX Software on all nodes. Refer to Upgrading the OPX Software.
Note
This step is required with every software upgrade.

7.3.2.6.3. Upgrading the OPX Software

To upgrade the CN5000 OPX Software, you first upgrade the management node then the remaining servers.

7.3.2.6.3.1. Assumption

You have downloaded the required software RPMs for upgrade.
You have stopped all standby FMs, followed by the primary FM.
You are logged in to the target Management Node.

7.3.2.6.3.2. Procedure

The following instructions provide the steps for upgrading the host software on the first management node and then the remaining nodes.

Note

If you plan to use a local repository and the TGZ package from the Cornelis Customer Center, download and extract the package to the management node.

Download and Install the OPX Software package:

Create a repository folder or confirm that the following already exists.
- For RHEL:
```
/etc/yum.repos.d/
```
- For SLES:
```
/etc/zypp/repos.d/
```
- For Ubuntu:
```
/etc/apt/sources.list.d/
```

Create a repository file.

For RHEL:

sudo vi /etc/yum.repos.d/cornelis-repository.repo
[Cornelis-Package]
name=Cornelis Repository
baseurl=file:///<file path to local folder where downloaded tar file has been extracted>
enabled=1
gpgcheck=1

For SLES:

sudo vi /etc/zypp/repos.d/cornelis-repository.repo
[Cornelis-Package]
name=Cornelis Repository
baseurl=file:///<file path to local folder where downloaded tar file has been extracted>
autorefresh=1  
enabled=1
gpgcheck=1

For Ubuntu:

sudo vi /etc/apt/sources.list.d/cornelis-repository.list
deb [trusted=yes] file:<file path to local folder where downloaded tar file has been extracted>/ ./

Import the GPG Key.

For RHEL and SLES:

rpm --import <file path to local folder where downloaded tar file has been extracted>/CN5000-Packages-Public-GPG-Key.asc

For Ubuntu:

sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/cornelis.gpg <file path to local folder where downloaded tar file has been extracted>/CN5000-Packages-Public-GPG-Key.asc

Clean and update the package Index.
- For RHEL:
```
sudo dnf clean all
sudo dnf makecache
```
- For SLES:
```
sudo zypper clean
sudo zypper refresh
```
- For Ubuntu:
```
sudo apt clean
sudo apt update
```
Caution
Installing meta packages will override existing packages. If multiple versions are desired (for example, GPU/non-GPU), additional components can be installed directly using the package manager, but not using the meta packages.
Determine the meta package you need to use for installation.
Note
Ubuntu uses dashes in place of underscores in the following list.
- cn5000_pkgs_non_gpu_meta - CPU-based installation
- cn5000_pkgs_cuda_meta - NVIDIA GPU installation
- cn5000_pkgs_rocm_meta - AMD GPU installation
Upgrade the package using one of the following commands.
- For RHEL:
```
sudo dnf install <META_PACKAGE_NAME>
```
- For SLES:
```
sudo zypper install <META_PACKAGE_NAME>
```
  Note
  If zypper prompts for confirmation, you can select the default to continue.
- For Ubuntu:
```
sudo apt install <META_PACKAGE_NAME>
```
  Note
  If you receive an error related to required dependencies that are not the latest versions available on Ubuntu, Cornelis recommends that you install and use aptitude in place of apt in the command.
  Alternately, you can use apt or apt-get with --mark-auto to manually install the dependencies.
Perform the following:
- For RHEL and SLES, continue to the next step to download, build, and install the kmod-opxs-kernel-updates packages.
- For Ubuntu, reboot the server to finalize the installation, then go to Step 13 to install the software to each remaining server.
  Note
  The opxs-modules-dkms packages are already installed with the Ubuntu meta package.

Download, build, and install the kmod-opxs-kernel-updates packages (RHEL and SLES only):

Copy the relevant RPM source file (src.rpm) to a local directory.
Important
- Ensure you are using the correct src.rpm file to build and install the version of the opx-kernel-updates relevant to your hardware configuration (AMD GPU, NVIDIA GPU, or CPU-only). Three opxs-kernel-updates RPM source files are provided, one each for AMD GPU, NVIDIA GPU, and CPU-only.
- RHEL Only: To rebuild the source RPM, you must install kernel-rpm-macros (for AMD GPUs) and kernel-abi-stablelists.
- SLES Only: You need to build unifdef before running rpmbuild:
```
cd /lib/modules/$(uname -r)/source/scripts
gcc -o unifdef unifdef.c
```

Build the source RPM.

rpmbuild --rebuild --define "_topdir <local_directory>" --define 'dist %{nil}' --target x86_64 --define 'kver $(uname -r)' <src.rpm_file>

Verify the RPMs are built by changing directories to the <local_directory>/RPMS/x86_64 directory and running ls -1.

Output should be:

For RHEL:

kmod-opxs-kernel-updates-<build_version>.x86_64.rpm
opxs-kernel-updates-devel-<build_version>.x86_64.rpm

For SLES:

kmod-opxs-kernel-updates-<build_version>.x86_64.rpm
opxs-kernel-updates-kmp-default <build_version>.x86_64.rpm

Upgrade the new RPMs.

For RHEL:

sudo dnf install kmod-opxs-kernel-updates-<build_version>.x86_64.rpm 
opxs-kernel-updates-devel-<build_version>.x86_64.rpm

For SLES:

zypper --no-gpg-checks install opxs-kernel-updates-devel-<build_version>.x86_64.rpm 
opxs-kernel-updates-kmp-default-<build_version>.x86_64.rpm

Reboot the primary node to apply changes.

Install the OPX Software to each remaining server:

Copy the Repository folder, the tar file (if using a local repository), and the <local_directory> containing the newly built RPMs, to each remaining server.
Important
The folders and files must be in the same directories as was in the management node.
Extract the TGZ package.
Upgrade the software on the remaining nodes using the commands in steps 6 and 11.
Reboot the rest of the nodes to apply changes.

7.3.2.6.4. Update the Fabric Manager opafm Configuration File

The opafm configuration file is designed so that any new parameters that might not appear in your old opafm.xml file will automatically be set to their defaults in the software. If you have customized your pre-upgrade opafm.xml file, it is recommended that you manually update your new opafm.xml file with the latest comments and defaults. This will aid you with future customizations and make comparisons with the opafm.xml in future upgrades easier.

Important

The opafm configuration file must be the same on all management nodes.

7.3.2.6.4.1. Assumption

You have upgraded the Fabric Manager, accepting all the defaults.

7.3.2.6.4.2. Procedures

Perform the following steps to transfer customizations into a new opafm.xml file:

Create a new opafm.xml using the /usr/share/opa-fm/opafm.xml file as a base for the new file by copying this read-only file to a temporary location to make the subsequent edits.
Update the new opafm.xml with the required customizations.
Use the new file from the previous step to replace the /etc/opa-fm/opafm.xml file.
Restart the Fabric Manager.
- If you previously upgraded the management node, type reboot and press Enter.
  Note
  If the Omni-Path Autostart was disabled, restart the primary Fabric Manager, followed by the standby Fabric Manager. Refer to Starting the Fabric Manager.
- If you only updated the opafm.xml file, type systemctl restart opafm and press Enter.
Copy the opafm.xml file to the standby Fabric Managers, replacing the /etc/opa-fm/opafm.xml file.
Restart the standby Fabric Managers.

7.3.2.6.5. Starting the Fabric Manager

This section provides instructions for starting the Fabric Manager.

Important

The Fabric Manager is always run from a node connected to the latest generation switch (for example, a CN5000 switch).

7.3.2.6.5.1. Prerequisites

Prior to starting the Fabric Manager, you must edit the opafm.xml file (located under /etc/opa-fm/) and ensure the following:

HFI 1, port 2 is enabled for fm0:
```
<Start>1</Start>  <-- 1 signifies enabled
<Name>fm0</Name> 
<Hfi>1</Hfi> 
<Port>2</Port>
```
Note
Also note that a single port SuperNIC appears as dual port in opainfo output.
CableInfoPolicy is set to None:
```
<CableInfoPolicy>None</CableInfoPolicy>
```

7.3.2.6.5.2. Procedure

To start the Fabric Manager, perform the following steps:

Log into the Fabric Manager system as root or as a user with root privileges.
Start the Fabric Manager:
```
systemctl start opafm
```
Verify that all the tasks are up and running:
```
systemctl status opafm
```
Note
The default configuration runs the Fabric Manager on port 2 of the first SuperNIC in the system.

Would you like to provide feedback? Just click here to suggest edits.

Cornelis Technical Documentation

7.3.2.6. Update the OPX Software

7.3.2.6.1. Prerequisites

Note

7.3.2.6.2. Update Sequence

7.3.2.6.1. Stopping the Fabric Manager

7.3.2.6.2. Resolving CUDA Dependency Errors for an OPX Software Upgrade

Procedures

Note

Note

Note

Note

7.3.2.6.3. Upgrading the OPX Software

7.3.2.6.3.1. Assumption

7.3.2.6.3.2. Procedure

Note

Caution

Note

Note

Note

Note

Important

Important

7.3.2.6.4. Update the Fabric Manager opafm Configuration File

Important

7.3.2.6.4.1. Assumption

7.3.2.6.4.2. Procedures

Note

7.3.2.6.5. Starting the Fabric Manager

Important

7.3.2.6.5.1. Prerequisites

Note

7.3.2.6.5.2. Procedure

Note

Search results