Speeding up processing data from millions of smart meters

As an important element of the Smart Grid, Advanced Metering Infrastructure (AMI) systems have been implemented and deployed throughout the world in the past several years. An AMI system connects millions of end devices (e.g., smart meters and sensors in the residential level) with utility control centers via an efficient two-way communication infrastructure. AMI systems are able to exchange substantial meter data and control information between utilities and end devices in real-time or near real-time. The major challenge our research was to scale ABB's Meter Data Management System (MDMS) to manage data that originates from millions of smart meters. We designed a lightweight architecture capable of collect ever-increasing large amount of meter data from various metering systems, clean, analyze, and aggregate the meter data to support various smart grid applications. To meet critical high performance requirements, various concurrency processing techniques were implemented and integrated in our prototype. Our experiments showed that on average the implemented data file parser took about 42 minutes to complete parsing, cleaning, and aggregating 5.184 billion meter reads on a single machine with the hardware configuration of 12-core CPU, 32G RAM, and SSD Hard Drives. The throughput is about 7.38 billion meter reads (206.7GB data) per hour (i.e., 1811TB/year). In addition, well-designed publish/subscribe and communication infrastructures ensure the scalability and flexibility of the system.