ENDMal: An anti-obfuscation and collaborative malware detection system using syscall sequences

Abstract

Malware obfuscation obscures malware into different versions, making traditional syntactic nature based detection ineffective. Furthermore, with the huge and exponentially growing number of malware samples, existing malware detection systems are either evaded by malware obfuscation, or overwhelmed by numerous malware samples. This paper proposes an anti-obfuscation, scalable and collaborative malware detection system—ENDMal. ENDMal identifies the program that behaves suspiciously in end-hosts and similarly between a group of suspicious programs in a wide area as malicious. We present the Iterative Sequence Alignment (ISA) method to defeat malware obfuscation. Instead of using complex behavior graph, we propose the Handle dependences and Probabilistic Ordering Dependence (HPOD) technology to represent the program behaviors. In addition, we design a novel information sharing infrastructure, RENShare, to collaboratively congregate the group characteristics of programs spreading over different network areas. Our experimental results show that ENDMal can detect unknown malwares much faster than the centralized detection system and is more effective than the existing distributed detection system.