Estimating the Number of Active Flows in a Data Stream over a Sliding Window

Abstract

A new algorithm is introduced to estimate the number of distinct flows (or connections) in a data stream. The algorithm maintains an accurate estimate of the number of distinct flows over a sliding window. It is simple to implement, parallelizes optimally, and has a very good trade-off between auxiliary memory and accuracy of the estimate: a relative accuracy of order 1/√m requires essentially a memory of order mln(n/m) words, where n is an upper bound on the number of flows to be seen over the sliding window. For instance, a memory of only 64kB is sufficient to maintain an estimate with accuracy of order 4 percents for a stream with several million flows. The algorithm has been validated both by simulations and experimentations on real traffic. It proves very efficient to monitor traffic and detect attacks.