Running the RoCE Adapter in NIC Mode

Edit: Some links no longer work.

Originally posted December 10, 2013 on AIXchange

A colleague sent me an interesting solution to a problem he was seeing with a PCI32 10GbE RoCE converged host bus adapter.

It was coming up in RDMA mode by default under VIOS 2.2.2.3 (which is AIX 6100-08-03-1339 under the covers), and the customer wanted it to run it in NIC mode:

            “The PCIe2 10 GbE RDMA Over Converged Ethernet (RoCE) Adapter was supported only on previous versions of the AIX operating system to use the Remote Direct Memory Access (RDMA) configuration mode. AIX 7 with 7100-02 or later supports the adapter that is configured in either the RDMA or the network interface card (NIC) configuration. The host bus adapter (HBA), which was not available in earlier versions of the AIX operating systems, manages which mode is enabled.

            “As of AIX 7 with 7100-02, the PCIe2 10 GbE RoCE Adapter can be configured to run in the NIC configuration. If you do not have the network-intensive applications that benefit from RDMA, then you can run the adapter in the NIC configuration.”

The preceding URL includes some instructions on moving from RDMA to NIC (or from NIC to RDMA), but also note the steps taken by my colleague in his situation:

            ISSUE: PCI card EC30 (PCIe2 10GbE RoCE Converged Host Bus Adapter) is presenting only as hba0 and roce0 and not providing ethernet over fibre devices such as entX. The intention is to use this card for use on a VIO server as a shared Ethernet adapter in NIC mode.

            SOLUTION: The stacktype has to be changed from aix_ib to ofed. The default for this card as installed this instance is aix_ib (Infiniband).

########################

#  VIEW CARD LOCATION  #

########################

# lscfg |grep “-C5-“

* hba0             U78C5.001.DQD02KZ-P2-C5-T1        PCIe2 10GbE RoCE Converged Host Bus Adapter (b315506714106104)

+ roce0            U78C5.001.DQD02KZ-P2-C5-T1-L0     PCIe2 10GbE RoCE Converged Network Adapter

########################

#  REMOVE ROCE DEVICE  #

########################

# rmdev -dl roce0

roce0 deleted

#############################

#  VIEW ATTRIBUTES OF HBA0  #

#     (note stack_type)     #

#############################

# lsattr -El hba0

bar0          0xfbf00000         Bus memory address 0 False

bar1          0xfc000000         Bus memory address 1 False

bar2          0x80000000         Bus memory address 2 False

busintr       0                  Bus interrupt level  False

busintrl      129536             Bus interrupt        False

devid         0xb315506714106104 Device ID            False

intr_priority 3                  Interrupt priority   False

rom_mem       0x80080000         ROM memory address   False

stack_type    aix_ib             RoCE Stack Type      True

#############################

#  CHANGE DEVICE ATTRIBUTE  #

#############################

# chdev -l hba0 -a stack_type=ofed

hba0 changed

########################

#  RUN CFGMGR          #

########################

# cfgmgr

#############################

#  VIEW ATTRIBUTES OF HBA0  #

#     (note stack_type)     #

#############################

# lsattr -El hba0

bar0          0xfbf00000         Bus memory address 0 False

bar1          0xfc000000         Bus memory address 1 False

bar2          0x80000000         Bus memory address 2 False

busintr       0                  Bus interrupt level  False

busintrl      129536             Bus interrupt        False

devid         0xb315506714106104 Device ID            False

intr_priority 3                  Interrupt priority   False

rom_mem       0x80080000         ROM memory address   False

stack_type    ofed               RoCE Stack Type      True

#########################

#  VIEW CARD LOCATION   #

# (note new ent devices #

#########################

# lscfg |grep “-C5-“

* hba0             U78C5.001.DQD02KZ-P2-C5-T1        PCIe2 10GbE RoCE Converged Host Bus Adapter (b315506714106104)

+ ent4             U78C5.001.DQD02KZ-P2-C5-T1-L1     RoCE Converged Network Adapter

+ ent5             U78C5.001.DQD02KZ-P2-C5-T1-L2     RoCE Converged Network Adapter

As noted, information on this issue can be found under PCIe2 10 GbE RoCE Adapter support:

            “The PCIe2 10 GbE RoCE Adapter is preconfigured to operate in the RDMA configuration mode. A network that uses RDMA is more complicated to set up than the NIC configuration mode, but provides better performance than the NIC mode for network-intensive applications. This mode is often helpful for network storage or high-performance computing.”