Enhancing intrusion detection in containerized services: Assessing machine learning models and an advanced representation for system call data

Abstract

Security is one of the most critical requirements for modern digital systems. As the paradigm shifts from attempting to develop fully secure systems to designing resilient strategies that detect, respond to, and recover from attacks, Intrusion Detection Systems (IDS) become indispensable. However, developing robust IDS that address sophisticated attacks—especially in scenarios such as Cloud services, IoT, edge computing, and microservices, remains a significant challenge. Among these, containerized services present unique security challenges due to their architecture, deployment methods, and reliance on shared resources. On the other hand, Machine Learning (ML) offers promising, but not yet fully understood, solutions to enable automated, scalable, and adaptive intrusion detection mechanisms. In this paper, we study the applicability of a ML-based approach to enhance intrusion detection in containerized services by training and testing various ML algorithms on system call data, a commonly used data type in intrusion detection. Furthermore, we propose a novel graph-based representation for system calls that preserves critical relationships and contextual information between system calls. With this improved representation, we achieve enhancements in intrusion detection performance, including an increase in detection rates by at least 193% for the tested vulnerabilities while maintaining false alarms at a safer threshold, below a mean of 0.4% to maximize attack identification while minimizing false alarms we also incorporate a post-processing phase using a sliding window technique. This work not only addresses the challenges of securing containerized environments but also provides a robust framework for leveraging machine learning to build next-generation IDS.