Expiration filter: Mining recent heavy flows in high-speed networks

Abstract

Mining recent heavy flows, which indicate the latest trends in high-speed networks, is vital to network management and numerous practical applications such as anomaly detection and congestion resolution. However, existing network traffic measurement solutions fall short of capturing and analyzing network traffic combined with temporal information, leaving us unaware of the real-time status of network streams, such as those undergoing congestion or attacks. This paper proposes the Expiration Filter (EF), which focuses on mining recent heavy flows, enhancing our understanding of the current behavioral patterns within the data. Given the skewness in real-world data streams, EF first filters out small flows to improve accuracy and tracks only flows with large volumes that have recently emerged. The EF also incorporates a dynamically self-cleaning mechanism to evict outdated records and free up memory space for new flows, thus fitting into the constrained on-chip space. Additionally, the adopted multi-stage design ensures the hardware implementation of EF in emerging programmable switches for line-rate processing. Hence, we provide detailed insights into implementing EF in programmable hardware under strict programming and resource constraints. Extensive experiments on real-world datasets demonstrate that EF outperforms the benchmarks in terms of flow size estimation, identifying top-k recent flows and detecting heavy hitters. All source codes are available at Github (https://github.com/hanyifansuda/Expiration-Filter).