Realtime Malicious Traffic Detection Targeted for TCP Out-of-Order Packets Based on FPGA

Abstract

Currently, with the increasing popularity of high-speed network, in order to protect the network environment, more and more companies start to explore how to efficiently detect malicious traffic. On the software side, traditional detection systems are usually based on CPU which will consume multi-core processing ability to handle huge network traffic. On the hardware side, current researches focus on using specific hardware to offload some functions such as string matching in malicious traffic detection. However, they cannot detect attack behaviors hidden in TCP out-of-order (OOO) packets well, which are very common in modern complex network environments. To deal with this problem, we present an FPGA-based realtime malicious traffic detection method especially to inspect TCP OOO packets. It employs two core function designs for efficient malicious traffic inspection: TCP OOO reassembly and hierarchical packet match. First, the TCP OOO packets are reassembled to in-order packets to prevent the omission check of malicious traffic. Second, we adapt a hierarchical packet match design which can not only detect the packet header and filter the matching traffic, but also has the ability to inspect the carried payload to further determine whether the traffic is benign or malicious. We use Xilinx Alveo U50 accelerator card as the implementation platform to achieve high speed detection. This paper aims to provide a full detection path and implement all the reassembly and inspection process within an FPGA board. We adapt Cisco TRex as the traffic generator to evaluate the system from detection throughput, resource utilization and power consumption. Compared with the CPU-based approaches, the experiment results show that our system has 485% detection throughput increase and 68% average power decrease.