Optimized Storage and Fast Retrieval of Large Monitoring Datasets without Compromising Granularity

Abstract

The adoption of low power, small footprint systems such as Hewlett Packard Moons hot cartridge servers massively increases the number of servers in cloud/farms implementations. Understanding problems, bottlenecks, and scaling of distributed applications running on such clusters requires the ability to replay the exhaustive data collected by monitoring systems. Current monitoring solutions make compromises, simplify (i.e. Destroy) the data over time or do not scale. Moreover, in the cloud model, server roles and assignments often change, making it mandatory to correlate monitoring data with higher level information such as task assignments known by scheduling software. We present an optimized and fast process to store and retrieve monitoring data, allowing access to all samples collected without any granularity loss and, at the same time, a generic mechanism to correlate with information from orchestrators.