Edit: Some links no longer work.
Originally posted December 10, 2013 on AIXchange
A colleague sent me an interesting solution to a problem he was seeing with a PCI32 10GbE RoCE converged host bus adapter.
It was coming up in RDMA mode by default under VIOS 2.2.2.3 (which is AIX 6100-08-03-1339 under the covers), and the customer wanted it to run it in NIC mode:
“The PCIe2 10 GbE RDMA Over Converged Ethernet (RoCE) Adapter was supported only on previous versions of the AIX operating system to use the Remote Direct Memory Access (RDMA) configuration mode. AIX 7 with 7100-02 or later supports the adapter that is configured in either the RDMA or the network interface card (NIC) configuration. The host bus adapter (HBA), which was not available in earlier versions of the AIX operating systems, manages which mode is enabled.
“As of AIX 7 with 7100-02, the PCIe2 10 GbE RoCE Adapter can be configured to run in the NIC configuration. If you do not have the network-intensive applications that benefit from RDMA, then you can run the adapter in the NIC configuration.”
The preceding URL includes some instructions on moving from RDMA to NIC (or from NIC to RDMA), but also note the steps taken by my colleague in his situation:
ISSUE: PCI card EC30 (PCIe2 10GbE RoCE Converged Host Bus Adapter) is presenting only as hba0 and roce0 and not providing ethernet over fibre devices such as entX. The intention is to use this card for use on a VIO server as a shared Ethernet adapter in NIC mode.
SOLUTION: The stacktype has to be changed from aix_ib to ofed. The default for this card as installed this instance is aix_ib (Infiniband).
########################
# VIEW CARD LOCATION #
########################
# lscfg |grep “-C5-“
* hba0 U78C5.001.DQD02KZ-P2-C5-T1 PCIe2 10GbE RoCE Converged Host Bus Adapter (b315506714106104)
+ roce0 U78C5.001.DQD02KZ-P2-C5-T1-L0 PCIe2 10GbE RoCE Converged Network Adapter
########################
# REMOVE ROCE DEVICE #
########################
# rmdev -dl roce0
roce0 deleted
#############################
# VIEW ATTRIBUTES OF HBA0 #
# (note stack_type) #
#############################
# lsattr -El hba0
bar0 0xfbf00000 Bus memory address 0 False
bar1 0xfc000000 Bus memory address 1 False
bar2 0x80000000 Bus memory address 2 False
busintr 0 Bus interrupt level False
busintrl 129536 Bus interrupt False
devid 0xb315506714106104 Device ID False
intr_priority 3 Interrupt priority False
rom_mem 0x80080000 ROM memory address False
stack_type aix_ib RoCE Stack Type True
#############################
# CHANGE DEVICE ATTRIBUTE #
#############################
# chdev -l hba0 -a stack_type=ofed
hba0 changed
########################
# RUN CFGMGR #
########################
# cfgmgr
#############################
# VIEW ATTRIBUTES OF HBA0 #
# (note stack_type) #
#############################
# lsattr -El hba0
bar0 0xfbf00000 Bus memory address 0 False
bar1 0xfc000000 Bus memory address 1 False
bar2 0x80000000 Bus memory address 2 False
busintr 0 Bus interrupt level False
busintrl 129536 Bus interrupt False
devid 0xb315506714106104 Device ID False
intr_priority 3 Interrupt priority False
rom_mem 0x80080000 ROM memory address False
stack_type ofed RoCE Stack Type True
#########################
# VIEW CARD LOCATION #
# (note new ent devices #
#########################
# lscfg |grep “-C5-“
* hba0 U78C5.001.DQD02KZ-P2-C5-T1 PCIe2 10GbE RoCE Converged Host Bus Adapter (b315506714106104)
+ ent4 U78C5.001.DQD02KZ-P2-C5-T1-L1 RoCE Converged Network Adapter
+ ent5 U78C5.001.DQD02KZ-P2-C5-T1-L2 RoCE Converged Network Adapter
As noted, information on this issue can be found under PCIe2 10 GbE RoCE Adapter support:
“The PCIe2 10 GbE RoCE Adapter is preconfigured to operate in the RDMA configuration mode. A network that uses RDMA is more complicated to set up than the NIC configuration mode, but provides better performance than the NIC mode for network-intensive applications. This mode is often helpful for network storage or high-performance computing.”