Anomaly Detection of System Logs Based on Natural Language Processing and Deep Learning

Abstract

System logs record the execution trajectory of the system and exist in all components of the system. Nowadays, the systems are deployed in a distributed environment and they generate logs which contain complex format and rich semantic information. Simple statistical analysis methods cannot fully capture log information for effective abnormal detection of software systems. In this paper, we propose to analyze the logs by combining feature extraction methods from natural language processing and anomaly detection methods from deep learning. Two feature extraction algorithms, Word2vec and Term Frequency-Inverse Document Frequency (TF-IDF), are respectively adopted and compared here to obtain the log information, and then one deep learning method named Long Short-Term Memory (LSTM) is applied for the anomaly detection. To validate the effectiveness of the proposed method, we compare LSTM with other machine learning algorithms, including Gradient Boosting Decision Tree (GBDT) and Naïve Bayes, the results show that LSTM can perform the best for anomaly detection of system logs with both of the two feature extraction methods, indicating that LSTM can capture contextual semantic information effectively in log anomaly detection and will be a promising tool for log analysis.