Tracer: enforcing mandatory access control in commodity OS with the support of light-weight intrusion detection and tracing

Abstract

Enforcing a practical Mandatory Access Control (MAC) in a commercial operating system to tackle malware problem is a grand challenge but also a promising approach. The firmest barriers to apply MAC to defeat malware programs are the incompatible and unusable problems in existing MAC systems. To address these issues, we start our work by analyzing the technical details of 2,600 malware samples one by one and performing experiments over two types of MAC enforced operating systems. Based on the preliminary studies, we design a novel MAC model incorporating intrusion detection and tracing in a commercial operating system, named Tracer, in order to disable malware on hosts while offering good compatibility to existing software and good usability to common users who are not system experts. The model conceptually consists of three actions: detecting, tracing and restricting suspected intruders. One novelty is that it leverages light-weight intrusion detection and tracing techniques to automate security label configuration that is widely acknowledged as a tough issue when applying a MAC system in practice. The other is that, rather than restricting information flow as a traditional MAC does, it traces intruders and restricts only their critical malware behaviors, where intruders represent processes and executables that are potential agents of a remote attacker. Our prototyping and experiments on Windows show that Tracer can effectively defeat all malware samples tested via blocking malware behaviors while not causing a significant compatibility problem.