Fast, memory-efficient traffic estimation by coincidence counting

We consider the problem of fast, estimation of flow rates in backbone network links with possibly millions of flows. Accurate flow rate estimation is necessary for network traffic management, network planning, measuring compliance to service level agreements, and network security. Ideally, a rate estimation scheme should have short estimation times with provable bounds on estimation error, be low in memory usage, and be easily implementable in hardware for operation at high speeds. We develop such a scheme, and achieve up to two orders of magnitude speed-up in estimation time over the previously proposed two-runs-based RATE scheme [Kodialam, M et al., 2004]. The speedups are achieved without a significant increase in memory usage, by using coincidences instead of runs. Counting coincidences has a higher processing overhead than detecting two-runs, but this higher overhead is not significant for a hardware implementation. We show that the proposed scheme is faster and more accurate than other recently proposed schemes such as ACCEL-RATE [Hao, F et al., 2004] and smart sampling [Duffield, N et al., 2004]. The faster estimation time of the new scheme has many benefits including quicker detection of incipient denial of service attacks. We prove bounds on the scheme's accuracy, memory needs, and also show that it performs well by simulations that use both synthetic and real traffic traces.