An Online Parsing Framework for Semistructured Streaming System Logs of Internet of Things Systems

Abstract

This article presents a novel log abstraction framework based on neural open information extraction (OpenIE) and dynamic word embedding principles. Though various log parsing frameworks are proposed in the literature, the existing frameworks are modeled on predefined heuristics or auto-regressive methodologies that work well in offline scenarios. However, these frameworks are less suitable for dynamic self-adaptive systems, such as the Internet of Things (IoT), where the log outputs have diverse contextual variations and disparate time irregularities. Therefore, it is essential to move away from these traditional approaches and develop a systematic model that can effectively analyze log outputs in real-time and increase the system up-time of IoT networks so that they are almost always available. To address these needs, the proposed framework used OpenIE along with term frequency/inverse document frequency (TF/IDF) vectorization for constructing a set of relational triples (aka triple-sets). Additionally, a dynamic pretrained encoder–decoder architecture is utilized to imbibe the positional and contextualized information in its resultant outputs. The adopted methodology has enabled the proposed framework to extract richer word representations with dynamic contextualization of time-sensitive event logs to enhance further downstream activities, such as failure prediction and prognostic analysis of IoT networks. The proposed framework is evaluated on the system event log traces accumulated from a long range wide-area network (LoRaWAN) IoT gateway to proactively determine the probable causes of its various failure scenarios. Additionally, the study also provided a comparative analysis of its mathematical representations with that of the current state-of-the-art (SOTA) approaches to project the advantages and benefits of the proposed model, particularly from its data analytics standpoint.