Extendable MQTT Broker for Feedback-based Resource Management in Large-scale Computing Environments

Abstract

High-performance computing (HPC) systems demand continuous monitoring to ensure efficient resource allocation and application performance. Recent studies indicate that real-time resource utilization monitoring can significantly improve the performance of dynamic scheduling algorithms. However, latency induced by protocol stack heavily impacts the effectiveness of dynamic scheduling. In this paper, we propose a novel monitoring system that implements the protocol stack on a Field-Programmable Gate Array (FPGA) and adopts a publish/subscribe (pub/sub) communication protocol. Specifically, by introducing an FPGA-based protocol stack, we substantially reduce the latency of protocol stack processing and enable the implementation of custom plugins at the L7 layer. Our experiments demonstrate that the proposed system effectively reduces protocol stack latency and, with the extensibility provided by user-defined plugins, offers great potential for a wide range of HPC monitoring and feedback applications.