When reviewing networking products, often the first value quoted in the specifications is the bandwidth, or the rate at which data can be sent or received, not including link overhead such as error-checking or clock bits. This value is usually the maximum bandwidth in units of gigabits per second (Gbps). For CN5000 Omni-Path, this value is 400 Gbps. Note here that for CN5000 products, it does not need to be stated if the value is for send or receive, as CN5000 uses a full-duplex link. In other words, CN5000 products can send and receive simultaneously and can do so at the same rates. Note here that the 400 Gbps rate is listed as a maximum bandwidth. This is because the effective bandwidth of a link is a function of the message size, or the size of the individual pieces of data being transmitted.

The maximum transmission unit (MTU) for CN5000 is 10240 bytes. This value represents the largest possible message that can be transmitted. Data transfers larger than this value must be split into multiple messages.

As the 81.92 Kib MTU is small in comparison to the maximum bandwidth (400 Gbps), a multiple, MTU-sized message can be sent in a single second. Assuming that the maximum bandwidth is possible with an MTU-sized message, five million MTU-sized messages can be transmitted each second. This is the message rate, or the rate at which messages are transmitted. If the message rate is held constant, but the message size is reduced by half, the effective bandwidth is reduced from the maximum value of 400 Gbps to 200 Gbps. Consequently, the message rate for large messages (greater than 128 bytes) does increase as the message size reduces; however, this effect is not linear. The message rate for small messages (less than 16 bytes) does not change with respect to message size, as the message rate is hardware limited.

These effects combine to produce differing effective bandwidth values for each message size. Note that this discussion assumes an ideal server. In real applications, the chipset, processor, and application itself can have significant effects on effective bandwidth.