High performance CDR processing with MapReduce

Abstract

A Call Detail Record (CDR) is a data record produced by telecommunication equipment consisting of detail of call transaction logs. It contains valuable information for many purposes of several domains such as billing, fraud detection and analytical purposes. However, in the real world, these needs face a big data challenge. Billions CDRs are generated every day and the processing systems are expected to deliver results in a timely manner. In our case, the system also has constraint that is running in limited computation resources. We found that our current production system was not enough to meet these needs. We had successfully analyzed the current system bottleneck and found the root cause. Based on this analysis, we designed and implemented a better performance system which is based on MapReduce and running on Hadoop cluster. This paper presents the analysis of previous system and the design and implementation of new system, called MS2. In this paper, we also provide empirical evidence demonstrating the efficiency and linearity of MS2. In a test case of telecommunication mediation system, our test has shown that MS2 reduces overhead by 44% and speedup performance by nearly twice compared to previous system. From benchmarking with several related technologies in large scale data processing, MS2 is also shown to perform better in case of CDR batch processing. Running on a cluster consists of eight core CPU and two conventional disks, MS2 is able to process 67,000 CDRs/second.