Applying Value-Based Deep Reinforcement Learning on KPI Time Series Anomaly Detection

Time series anomaly detection has become more critical with the rapid development of network technology, especially in cloud monitoring. We focus on applying deep reinforcement learning (DRL) in this question. It is not feasible to simply use the traditional value-based DRL method because DRL cannot accurately capture important time information in time series. Most of the existing methods resort to the RNN mechanism, which in turn brings about the problem of sequence learning. In this paper, we conduct progressive research work on applying value-based DRL in time series anomaly detection. Firstly, because of the poor performance of traditional DQN, we propose an improved DQN-D method, whose performance is improved by 62% compared with DQN. Second, for RNN-based DRL, we propose a method based on improved experience replay pool (DRQN) to make up for the shortcomings of existing work and achieve excellent performance. Finally, we propose a Transformer-based DRL anomaly detection method to verify the effectiveness of the Transformer structure. Experimental results show that our DQN-D can obtain performance close to RNN-based DRL, DRQN and DTQN perform well on the dataset, and all methods are proven effective.