Parallel Processing of Burst Detection in Large-Scale Document Streams and Its Performance Evaluation

Abstract

Online documents on the Internet are represented as a document stream because the documents have a temporal order. This has resulted in numerous studies on extracting a frequent phenomenon (involving keywords, users, locations etc.) known as a burst. Recently, with the growth of interest in social media, the number of documents created on the Internet has increased exponentially. Therefore, the speed-up of burst detection in a large-scale document stream is one of the most important challenges. In this paper, we propose a novel parallelization method for the parallel processing of Kleinberg’s burst detection algorithm in a large-scale document stream. Specifically, we present a technique to combine the inter-task parallelization model with the intra-task parallelization model. This combination can achieve seamless dynamic load balancing and detect bursts in a large-scale document streams in memory.