ANOMALY DETECTION IN SYSTEM LOGS IN THE SPHERE OF DIGITAL ECONOMY

Abstract

Anomaly detection in log file analysis is a method of automatically monitoring log files to identify suspicious activities. It plays a major role in the management of modern distributed large-scale systems. The detection of anomalies is a critical issue for data-driven systems in the digital economy. The real objective of a system log is to record the state of the system, its execution trajectory, and the important events at different critical points. System log data is a valuable and meaningful resource for understanding the status of system and performance problems; therefore, these logs are an extremely useful source for online monitoring and detection of anomalies. Simple statistical analytical techniques cannot fully capture log information for system detection of effective anomalies. In this paper, we introduce an approach of analysing the logs by combining a method of feature extraction with an anomaly detection algorithm from deep learning. For feature extraction, word2vec is used and after that, a deep autoencoder model with Long Short-Term Memory (LSTM) units is used for anomaly detection. In this process several techniques are applied to data ie principal component analysis (PCA) for dimension reduction, gaussian multivariate normal distribution to normally distributed data using mean and covariance. After detecting anomalies, the logs are further classified into different web attacks ie brute force, port scanning, sql injection and file inclusion are visualised with different graphs in the results section. The experimental findings show the effectiveness of the proposed anomaly detection learning algorithm.