5.4.1. Security Configuration
5.4.1.1. Securing the Default vFabric
Refer to Securing the Default vFabric.
5.4.1.2. Securing the Admin vFabric
Refer to Secure the Admin vFabric.
5.4.1.3. Securing Hosts
You can secure hosts by disabling the ability to send Management Datagrams (MAD) packets (a standard message format for manager–agent communication) on switches and switch ports. Refer to Disabling Ports.
Note
You can also secure vFabrics by securing all host nodes that are NOT Admin Nodes.
This is accomplished by creating the AdminNodes device groups file in /etc/opa-fm/dgs and adding one or more management host nodes to the group. Then, disable AllMgmtAllowed and enable AdminNodes under the Admin partition in the opafm_pp.xml file.
For detailed instructions, see Secure the Admin VFabric.
5.4.1.3.1. Disabling Ports
Note
The CN5000 Switch is also referred to as an Edge Switch.
You can secure one or more hosts by deactivating switch ports using the port disable <PORT_NAME> command.
The following example provides instructions for disabling a single port (host7).
Determine to which switch and switch port
host7is connected.host7is attached to switchedge002on port23.View
host7to see that MAD packets are allowed (Mgmt=True).[root@host1 ~]# ssh host7 Last login: Thu Aug 15 10:18:47 2024 from 192.168.0.200 [root@host7 ~]# opainfo | grep Mgmt LCRC Act: 14-bit En: 14-bit,16-bit,48-bit Mgmt: TrueThis host node is not secure. With management allowed, all commands can be run from
host7.(OPTIONAL) Run a command to find the LID information from every host in the fabric and a command to view all the links.
Note
The following example shows partial results.
[root@host7 ~]# opaextractlids Getting All Node Records... Done Getting vFabric Records 0x0011750101743e91;1;FI;rh212 hfi1_0;0x0001 0x0011750102703c47;0;SW;edge001;0x0002 0x001175010174428a;1;FI;rh216 hfi1_0;0x0003 0x0011750101743ed0;1;FI;rh217 hfi1_0;0x0004 0x00117501017443a8;1;FI;rh218 hfi1_0;0x0005
Since management is allowed, these commands produced the desired results.
Go to the switch
edge002.[root@host7 ~]# ssh admin@edge002 admin@edge002's password: Welcome to the Cornelis CN5000 Switch CLI. Type 'list' for the list of commands.
View the status of port 23 (
cable23) usingport show.Note
Ports are named Cable01 – Cable48 on CN5000 Switches (also called Edge Switches) and as Leaf modules in Director Class Switches.
Edge-> port show Cable23 Cable23 FM_ENABLED: ENABLED
The port is Enabled (default).
Disable port 23, then bounce the port.
Bouncing the port is momentarily disruptive but only to that switch port and attached host.
Edge-> port disable Cable23 Edge-> port bounce Cable23 Cable23 FM_ENABLED: DISABLED Disabled port Cable23! Enabled port Cable23!
View the status of port 23.
Edge-> port show Cable23 Cable23 FM_ENABLED: DISABLED
(OPTIONAL) Run the following commands to see how the disabled port appears in output:
Exit out of the switch and look at
Mgmtwithopainfocommand onhost7.Edge-> exit Connection to edge002 closed by remote host. Connection to edge002 closed. [root@host7 ~]# opainfo | grep Mgmt LCRC Act: 14-bit En: 14-bit,16-bit,48-bit Mgmt: FalseManagement is now set to False, so Management Datagrams (MAD) packets are not allowed. This is more secure.
Run some opa commands to the Fabric:
[root@rh227 ~]# opareport -o lids Getting All Node Records... Processed 0 of 41 Nodes... SA SA PortInfo query Failed: FPROTECTION
Most fabric commands are not allowed from a secured host node by the Subnet Administrator (SA).
To enable a port:
Run
port enableto enable the switch port 23.Edge-> port enable Cable23 Cable23 FM_ENABLED: ENABLED Disabled port Cable23! Enabled port Cable23!
View the status of the port.
Edge-> port show Cable23 Cable23 FM_ENABLED: ENABLED
5.4.1.4. Setting up Predefined Topology Verification Security
Note
Verify the cluster before enabling Predefined Topology Verification feature.
Create a predefined topology input configuration file using one of the following commands:
opaxlattopology
opagentopology
opareport
Edit the configuration file using one of the following methods:
Node GUIDs and Port Numbers
Node Descriptions and Port Numbers
For details, refer to Predefined Topology Verification Security
5.4.1.5. Replacing a Node with Predefined Topology Verification
This section describes how to replace a SuperNIC when using the Predefined Topology Verification feature. For more information, refer to Predefined Topology Verification.
For topology files based on node descriptions, the SuperNIC replacement process is straightforward: update the node description of the replacement node prior to adding to the cluster. No restart of FM is required, and the node will pass verification.
For topology files based on node GUIDs and port numbers, an FM Restart is required.
The node to be replaced should have its node description updated to the correct name prior to adding the node to the cluster.
Modify the input topology configuration file, replacing the old node GUID with the new node GUID. An alternative way to find the correct GUIDs to be replaced is to search the topology file for the replacement node description (assuming unique node descriptions).
Add the replacement node back to the cluster and restart all FMs. The newly replaced node should activate. If it does not, examine the logs for warnings indicating why the node did not activate; it may have been quarantined due to other reasons.
5.4.1.6. Setting User Query Permissions
Cornelis recommends that you set MgmtAllowed to Disabled for all nodes that are not management nodes. For more information, refer to User Query Permissions.