Extracting Overlay Invariants of Distributed Systems for Autonomic System Management

Abstract

Many large-scale distributed systems have been built with great complexity to run Internet services. Due to the heterogeneity and dynamics of complex systems, it is very difficult to characterize their behavior precisely for system management. While we collect large amount of monitoring data from distributed systems as system observables, it is hard for us to interpret the data without constructing reasonable system models. Our previous work proposed algorithms to extract invariants from monitoring data to profile complex systems. However, such invariants are extracted between pair wise system measurements but not among multiple measurements. Based on minimal redundancy maximal relevance subset selection and least angle regression, this paper proposes an efficient algorithm to automatically extract overlay invariants from the layer of pair wise invariant networks. The overlay invariants link separated pair wise invariant subnets and enable us to support many system management tasks such as fault detection and capacity planning. Experimental results from synthetic data and real commercial systems are also included to demonstrate the effectiveness and efficiency of our algorithm.