PM/Ethernet-kRMA: a high performance remote memory access facility using multiple gigabit ethernet cards

Abstract

This paper proposes a high performance communication facility using multiple commodity network interface cards (NICs). Called PM/Ethernet-kRMA, it is NIC-hardware-independent and provides (k)ernel-level Remote Memory Access (kRMA) on multiple NICs. The PM/Ethernet-kRMA communication protocol is processed on the host processor, and the protocol handler accesses user data space directly from the kernel, and then transfers the data to network using existing network device drivers. This protocol provides one-copy communication between user memory spaces on kernel. The PM/Ethernet-kRMA is implemented using the PM/Ethernet, one of the communication facilities of the SCore Cluster system software on Linux. The PM/Ethernet uses the Network Trunking technique, which provides message communication using multiple NICs. Existing protocols, such as TCP/IP, can be used on the PM/Ethernet-kRMA as well as the PM/Ethernet. We have evaluated the PM/Ethernet-kRMA using 2-node single Xeon 2.4GHz processor machines with three Intel PRO/ 1000 XTs and one Broadcom 5701 based Gigabit Ethernet NICs on each node. Network Trunking provides 420 MB/s of communication bandwidth using four Gigabit Ethernet NICs. PM/Ethernet-kRMA using four Gigabit Ethernet NICs, in contrast, provides 487 MB/s of bandwidth which is 97.4% of hardware-level bandwidth (500 MB/s).