Log2graphs: An Unsupervised Framework for Log Anomaly Detection with Efficient Feature Extraction

Abstract

In the era of rapid Internet development, log data has become indispensable for recording the operations of computer devices and software. These data provide valuable insights into system behavior and necessitate thorough analysis. Recent advances in text analysis have enabled deep learning to achieve significant breakthroughs in log anomaly detection. However, the high cost of manual annotation and the dynamic nature of usage scenarios present major challenges to effective log analysis. This study proposes a novel log feature extraction model called DualGCN-LogAE, designed to adapt to various scenarios. It leverages the expressive power of large models for log content analysis and the capability of graph structures to encapsulate correlations between logs. It retains key log information while integrating the causal relationships between logs to achieve effective feature extraction. Additionally, we introduce Log2graphs, an unsupervised log anomaly detection method based on the feature extractor. By employing graph clustering algorithms for log anomaly detection, Log2graphs enables the identification of abnormal logs without the need for labeled data. We comprehensively evaluate the feature extraction capability of DualGCN-LogAE and the anomaly detection performance of Log2graphs using public log datasets across five different scenarios. Our evaluation metrics include detection accuracy and graph clustering quality scores. Experimental results demonstrate that the log features extracted by DualGCN-LogAE outperform those obtained by other methods on classic classifiers. Moreover, Log2graphs surpasses existing unsupervised log detection methods, providing a robust tool for advancing log anomaly detection research.