Tuning Logstash Garbage Collection for High Throughput in a Monitoring Platform

Abstract

The collection and aggregation of monitoring data from distributed applications are an extremely important topic. The scale of these applications, such as those designed for Big Data, makes the performance of the services responsible for parsing and aggregating logs a key issue. Logstash is a well-known open source framework for centralizing and parsing both structured and unstructured monitoring data. As with many parsing applications, throttling is a common issue due to the incoming data exceeding Logstash processing ability. The conventional approach for improving performance usually entails increasing the number of workers as well as the buffer size. However, it is unknown whether these approaches might tackle the issue when scaling to thousands of nodes. In this paper, by profiling Java virtual machine, we optimize Garbage Collection in order to fine tune a Logstash instance in DICE monitoring platform to increase its throughput. A Logstash shipper simulation tool was developed to transfer simulated data to the Logstash instance. It is capable of simulating thousands of monitored nodes. The obtained results show that with our suggestion of minimizing Garbage Collection impact, the Logtash throughput increases considerably.