Soft real-time GPRS traffic analytics for commercial M2M communications using spark

Commercial applications of wireless sensor networks, also known as machine-to-machine (M2M) communications, feature hundreds of thousands or even millions of devices. These M2M applications often rely on cellular networks like GSM that were not designed with such use cases in mind. Based on our first-hand experience at a large provider of M2M communications solutions, there is a need for soft real-time traffic analytics solutions to help engineers monitor and manage the millions of devices deployed in these M2M applications. We present a solution for soft real-time GPRS traffic analytics built on Apache Spark, a framework for distributed in-memory computing. The proposed solution captures GPRS traffic, processes it, and decorates it with details about the devices, networks, and M2M applications. It then computes a whole array of statistics that are presented in charts and maps on a live Web application dashboard, or may be fed to other systems for data mining. In a series of experiments, previously captured GPRS traffic from real-life commercial M2M applications is played back to the traffic analytics solution at different rates, and is processed on clusters of varying size. Results show that our solution handles GPRS traffic rates of 3,333 packets/sec, which are 2X the rates of an M2M application with close to one million devices, with a latency below one minute on a Spark cluster with four m1.large slave instances in Amazon EC2 at a cost of $7,665/year. These costs can be reduced to approx. $700/year by bidding on SPOT instances.