Exploratory Data Analysis for Failure Detection and Isolation in Complex Systems

Abstract

Failure detection and isolation (FDI) is a crucial step in diagnostics and is quickly shifting to towards using analytical techniques such as machine learning and deep learning, rather than traditional rules-based approaches. This is partially due to the availability of sensor systems, hardware and networking allowing for a vast collection and processing of data. However, this information is prone to issues such as noise, corruption, poor formatting and recording practices. In most cases, a diagnostics project may stall midway due to late discovery of these problems. This paper proposes exploring the data beforehand, to locate issues in the data and/or optimize data quality to maximize performance or explain possible performance loss. Various techniques such as data visualization, statistical analysis and feature importance are mentioned. Most importantly, a domain knowledge set is to be integrated with such correlation-based methods to ensure that data quality decisions are made with understanding of the system. The limitations of such analysis including scalability and interpretation issues are discussed as well, leading to proposals of possible future paths to improvement such as sensor fusion and AI-based recommendations.