Time Series Anomaly Detection from a Markov Chain Perspective

Abstract

This paper introduces a new method for the pattern-wise anomaly detection problem, which aims to find segments whose behaviors are different from the rest of the segments in the time series (as opposed to finding a single data-point in classic anomaly detection problems). An important motivation for studying this problem is to find anomalies whose data-points are within the normal range but they create an unusual pattern. To this end, normal characteristics of the data are found by clustering the overlapping subsequences of the training dataset and analyzing their orders by Markov chains. The trained model is used to assess how well the testing dataset suits the baseline behavior. The designed anomaly detection framework is capable of discovering unusual patterns in both streaming data (online) and stored data (offline). The performance of the methodology is evaluated by applying it to three datasets from different fields: a medical dataset (electrocardiogram), a utility usage dataset, and a New York City taxi demand dataset. The detected anomaly in the medical data agrees with the results of the studies in the literature. A domain expert confirmed the accuracy of the results for the utility usage data, and the anomalies of the New York City taxi demand data referred to major US holidays.