Distributed Hayabusa: Scalable Syslog Search Engine Optimized for Time-Dimensional Search

Abstract

Network administrators usually collect and store logs generated by servers, networks, and security appliances so that when network trouble and/or security incidents occur, they can identify the source of the problem by investigating the contents of the logs. The size of the system needed to store and search the log messages tends to increase as the size of the managed network becomes large. A fast log storage and search system called Hayabusa was previously proposed that optimizes a time-dimensional search operation. In this paper, we propose a simple distributed system that adds scalability to the existing Hayabusa system. The evaluation results show that the Distributed Hayabusa system consisting of 10 servers (with multiple worker processes on each server) is 36 times faster than a standalone Hayabusa system. The time required to perform a full-text search over 14.4 billion data records is only about 7 s, which is sufficiently low for the daily operations of administrators managing a very-large-scale network.