CD-SR: A Real-time Anomaly Detection Framework for Continuous Concept Drift

Abstract

System administrators need to monitor various metrics (network traffic, NTP offset, etc.) of their internal services in real-time as a way to determine whether anomalies occur in the system. Traditional Spectral Residual (SR) anomaly detection methods do not take into account the interference of certain human factors (e.g., changes in personal preferences) in certain scenarios, i.e., concept drift. In these scenarios, the accuracy of anomaly detection is bound to be affected. In order to guarantee the availability and stability of network services, we propose an intelligent and pervasive anomaly detection strategy, CD-SR. First, we use the traditional SR model and the SVM method to train the time series that have not drifted to determine the threshold value. Then, to solve the problem of the pervasiveness of application scenarios, we use a drift detection model to find the time series where concept drift occurs. Finally, the sequence where the concept drift occurs is imported into the drift adaptation model to complete the replacement of the old and new concepts, the data is processed in real-time, and the replaced data is detected again in the detection model for anomalies. In the experimental stage, we obtained several data metrics using the cloud platform system built by Openstack, and by comparing several mainstream anomaly detection algorithms, our method obtained superior results.