Online Log Analysis(OLA) for Malicious User Activities

Abstract

Efficient log analysis involves collecting, evaluating, and managing raw data from computer-generated records. As security vulnerabilities increase, the analysis of logs has become vital and crucial in multidisciplinary domains. Maintaining and analyzing the log is a pivotal part of every organization as tons of logs are generated every millisecond. However, anomaly detection and log parsing addressed so far, rely on a time-consuming training algorithm based on a Machine Learning framework. The proposed method detects anomalies from real-time data generated from the data centre without the need for a training algorithm. Detection and visualization of malicious activities are done by Elasticsearch, Logstash, and Kibana (ELK) framework. The process of shipping, parsing, indexing, and anomaly detection is carried out using an unsupervised machine learning algorithm which gives a clear inference to detect bots and perform unique log session classification. A real-time Apache HTTP Server log is accessed and anomalous behavior is identified based on the incoming requests. Experiments on real-time data show that 13.76% of anomalies are detected on per weekly basis.