HPTCollector: high-performance telemetry collector

Abstract

Network telemetry plays a pivotal role in understanding and optimizing underlying network infrastructures by facilitating essential operations like troubleshooting and traffic load balancing. However, real-time processing of network packets, especially at speeds of 100 Gbps or more, presents significant challenges due to the uncoordinated processing performance between kernel and user-space applications. This study introduces high-performance telemetry collector (HPTCollector) aims at harmonizing the processing activities of kernel and user-space applications, thereby enhancing the performance of network telemetry systems. HPTCollector demonstrates exceptional adaptability and efficiency, achieving remarkable throughput rates. Specifically, our mechanism can process up to 31 million packets per second using just 12 CPU cores in user-space, an achievement made possible through parallel packet processing techniques. This capability ensures robust support for network telemetry processing at collector for network infrastructures with bandwidth of 350 Gbps and 2.03 Tbps, MTU size of 1500 and 9000 respectively. This breakthrough not only showcases the potential of our proposed mechanism but also sets a new benchmark in network telemetry collector performance.