Implementation of an efficient RDMA mechanism tightly coupled with a TCP/IP offload engine

Abstract

We develop a hybrid TCP/IP offload engine (hybrid TOE) that processes TCP/IP via hardware/software coprocessing based on an FPGA and a general-purpose embedded processor. We also develop an efficient remote direct memory access (RDMA) mechanism that is tightly coupled with the hybrid TOE. In this mechanism, the hybrid TOE performs CRC calculations using hardware modules and supports zero-copy data transmission; the host CPU simply generates and processes RDMA protocol headers. By using the hybrid TOE and the RDMA mechanism, computer systems can achieve good network performance with very low CPU utilizations, and thus they can be expected to show a great improvement in overall performance. In experiments on a gigabit Ethernet network, although the embedded processor operated with a 300 MHz core clock, which was one-seventh the speed of the host CPUpsilas clock, the hybrid TOE showed a minimum latency of 17.4 mus and a maximum bandwidth of 736 Mbps. The RDMA mechanism exhibited a minimum latency of 20.6 mus and a maximum bandwidth of 642 Mbps. Most importantly, the hybrid TOE and the TOE-based RDMA mechanism gave CPU utilizations of less than 5.6% and 8.4%, respectively-approximately one-tenth the utilizations when TCP/IP and TCP/IP-based RDMA were processed by the host CPU.