DESK: Distributed Observability Framework for Edge-Based Containerized Microservices

Modern information technology (IT) infrastructures are becoming more complex to meet the diverse demands of emerging technology paradigms such as 5G/6G networks, edge, and internet of things (IoT). The intricacy of these infrastructures grows further when hosting containerized workloads as microservices, resulting in the challenge to detect and troubleshoot performance issues, incidents or even outages of critical use cases like industrial automation processes. Thus, fine-grained measurements and associated visualization are essential for operation observability of these IT infrastructures. However, most existing observability tools operate independently without systematically covering the entire data workflow. This paper presents an integrated design for multi-stage observability workflows, denoted as DistributEd obServability frameworK (DESK). The proposed framework aims to improve observability workflows for measurement, collection, fusion, storage, visualization, and notification. As a proof of concept, we deployed the framework in a Kubernetes-based testbed to demonstrate the successful integration of various components and usability of collected observability data. We also conducted a comprehensive study to determine the caused overhead by DESK agents at the reasonably powerful edge node hardware, which shows on average a CPU and memory overhead of around 2.5 % of total available hardware resource.