Circumventing Server Bottlenecks: Indirect Large-Scale P2P Data Collection

Abstract

In most large-scale peer-to-peer (P2P) applications, it is necessary to collect vital statistics data - sometimes referred to as logs - from up to millions of peers. Traditional solutions involve sending large volumes of such data to centralized logging servers, which are not scalable. In addition, they may not be able to retrieve statistics data from departed peers in dynamic peer-to-peer systems. In this paper, we solve this dilemma through an indirect collection mechanism that distributes data using random network coding across the network, from which servers proactively pull such statistics. By buffering data in a decentralized fashion with only a small portion of peer resources, we show that our new mechanism provides a "buffering" zone and a "smoothing" factor to collect large volumes of statistics, with appropriate resilience to peer dynamics and scalability to a large peer population.