A Transnational Multi-cloud Distributed Monitoring Data Integration System

Abstract

The complexity of many IT services and facilities has been continuously increasing, and the complexity of related monitoring systems and the difficulty of managing it are also growing rapidly. The integration and analysis of the time-series data acquired from monitoring systems are based on intelligent operation and maintenance. Due to the complex deployment of IT service and its infrastructure, as well as the large scale and high frequency of monitored indicators, the monitoring service and the integration of monitored data are further complicated. In the meantime, monitoring data is featured by low-value density, large volume, high requirements for real-time performance and reliability, complex process of transforming the time series data, which brings great challenges for the existing data integration systems. This paper proposes a distributed monitoring data integration system. The system achieves the efficient and reliable integration of time series monitoring data through a lightweight distributed architecture. Different methods of distributed scheduling are adopted by the system to achieve the elastic scaling of integrated computing power and adjust the load capacities of upstream and downstream time-series databases. The effectiveness of the designed system is verified in a data integration scenario from Prometheus/VictoriaMetrics to InfluxDB.