HCMonitor: An Accurate Measurement System for High Concurrent Network Services

Abstract

As user-interactive services grow explosively in datacenters, latency has become one of the most deciding factors on user experience. Therefore, estimating the latency and detecting anomalies from the expected latency is essential to evaluate services’ performance. Although many existing tools have been used widely, their estimation methods can be divided into two categories. First, the traffic-sample-based approaches sample the network traffic for accelerating the estimation rather than measure every response time. Second, the full-traffic-based approaches, such as tcpdump and wrk, analyze data from kernel and leave the latency computation to the client-side. In this paper, we attempt to compute the applications’ server-side latency for every request in real-time, and eliminate kernel processing delay. We propose a system named HCMonitor. It monitors all the traffic by switch mirroring, which results in high throughput and more accuracy in server-side latency estimation. The latency measurement is transparent to network services and can be displayed in real time. Our evaluations show HCMonitor obtains higher throughput than tcpdump by over 1000 times. Compared to wrk, the tail latency accuracy estimated by HCMonitor shows a promotion by up to 72%~76% in high concurrent network, by eliminating delay produced by packet transfer, kernel network stack and packets queuing on client side.