Hiroshi Abe, K. Shima, Y. Sekiya, Daisuke Miyamoto, Tomohiro Ishihara, Kazuya Okada
{"title":"Hayabusa:简单快速的全文搜索引擎,用于海量系统日志数据","authors":"Hiroshi Abe, K. Shima, Y. Sekiya, Daisuke Miyamoto, Tomohiro Ishihara, Kazuya Okada","doi":"10.1145/3095786.3095788","DOIUrl":null,"url":null,"abstract":"In this study, we introduce a simple and high-speed search engine for large-scale system logs, called Hayabusa. Hayabusa uses SQLite, standard lightweight database software with GNU Parallel and general Linux commands, such that it can run efficiently without complex components. Network administrators can use Hayabusa to accumulate and store log information at high speeds and to search the logs quickly. In our experiments, Hayabusa required only 8 seconds to convert 1.2 M log messages into a database file. Moreover, Hayabusa required only 5 seconds to search a keyword from 1.7 billion records. Hayabusa achieved high-performance search speed in a stand-alone environment without a complex distributed environment. Compared with the distributed environment, Spark, the proposed stand-alone Hayabusa was approximately 27 times faster.","PeriodicalId":209819,"journal":{"name":"Proceedings of the 12th International Conference on Future Internet Technologies","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2017-06-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":"{\"title\":\"Hayabusa: Simple and Fast Full-Text Search Engine for Massive System Log Data\",\"authors\":\"Hiroshi Abe, K. Shima, Y. Sekiya, Daisuke Miyamoto, Tomohiro Ishihara, Kazuya Okada\",\"doi\":\"10.1145/3095786.3095788\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"In this study, we introduce a simple and high-speed search engine for large-scale system logs, called Hayabusa. Hayabusa uses SQLite, standard lightweight database software with GNU Parallel and general Linux commands, such that it can run efficiently without complex components. Network administrators can use Hayabusa to accumulate and store log information at high speeds and to search the logs quickly. In our experiments, Hayabusa required only 8 seconds to convert 1.2 M log messages into a database file. Moreover, Hayabusa required only 5 seconds to search a keyword from 1.7 billion records. Hayabusa achieved high-performance search speed in a stand-alone environment without a complex distributed environment. Compared with the distributed environment, Spark, the proposed stand-alone Hayabusa was approximately 27 times faster.\",\"PeriodicalId\":209819,\"journal\":{\"name\":\"Proceedings of the 12th International Conference on Future Internet Technologies\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2017-06-14\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"2\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Proceedings of the 12th International Conference on Future Internet Technologies\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1145/3095786.3095788\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Proceedings of the 12th International Conference on Future Internet Technologies","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1145/3095786.3095788","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Hayabusa: Simple and Fast Full-Text Search Engine for Massive System Log Data
In this study, we introduce a simple and high-speed search engine for large-scale system logs, called Hayabusa. Hayabusa uses SQLite, standard lightweight database software with GNU Parallel and general Linux commands, such that it can run efficiently without complex components. Network administrators can use Hayabusa to accumulate and store log information at high speeds and to search the logs quickly. In our experiments, Hayabusa required only 8 seconds to convert 1.2 M log messages into a database file. Moreover, Hayabusa required only 5 seconds to search a keyword from 1.7 billion records. Hayabusa achieved high-performance search speed in a stand-alone environment without a complex distributed environment. Compared with the distributed environment, Spark, the proposed stand-alone Hayabusa was approximately 27 times faster.