Dynamic Adaptive Mechanism Design and Implementation in VSS for Large-Scale Unified Log Data Collection

Abstract

This paper studies the collection of large-scale log data of information system and puts forward a dynamic adaptive mechanism for large-scale unified log data collection. Furthermore, we design and implement our method for pan-government industry safety operation management platform. The data flow processing architecture based on message queue is adopted to realize the decoupling of log collection, log processing and log reporting. The traffic peak clipping technology of message queue is adopted to ensure the safety and reliability of log transmission. According to the characteristics of log traffic, a design mode supporting dynamic adjustment of consumption group is proposed to meet the high-performance requirements of the system. The whole system can meet the centralized analysis, security threat perception and intelligent analysis of various security data. Meanwhile, we analyzed and compared with the traditional open-source log collection technology, our proposed method and system has clear advantages.