DIP

Abstract

Certain classes of log analytical models, such as those for log anomaly detection, require as inputs sequences of parsed log messages in which the message tokens that belong to the template of the message are indicated. For this reason, it is common for such a model to employ a log parser, a program that detects the template of each message in a log file. It has been shown that even the most accurate log parsers in the literature fail to achieve high accuracy at detecting the templates of messages from certain systems' log files. This paper presents DIP, a tree-based log parser. The primary methodological innovation of DIP lies in the mechanism it uses to determine whether pairs of very similar messages have the same template. While many existing parsers only consider the percentage of matching tokens between two similar messages in determining whether they have the same template, DIP considers in addition the actual tokens at which the two messages disagree, deeming a pair of similar messages to have the same template if and only if each of those tokens satisfies one in a certain set of three conditions. Our experimental results show that DIP can achieve an average accuracy that is superior to that obtained by each of the 13 parsers tested in a 2019 survey study on log parsers. Furthermore, we give evidence that it achieves this high accuracy without compromising in terms of runtime.