A scalable, non-parametric anomaly detection framework for Hadoop

Abstract

In this paper, we present a scalable and practical problem diagnosis framework for Hadoop environments. Our design features a decentralized approach based on hierarchical grouping and a novel non-parametric diagnostic mechanism. We evaluate our framework under various Hadoop workloads. The experimental results show that our design outperforms traditional methods significantly in the context of complex anomaly patterns and high anomaly probability.