Probing the Limits of Virtualized Software Protection

Abstract

Virtualization is becoming a prominent field of research not only in distributed systems, but also in software protection and obfuscation. Software virtualization has given rise to advanced techniques that may provide intellectual property protection and anti-cloning resilience. We present results of an empirical study that answers whether integrity of execution can be preserved for process-level virtualization protection schemes in the face of adversarial analysis. Our particular approach considers exploits that target the virtual execution environment itself and how it interacts with the underlying host operating system and hardware. We give initial results that indicate such protection mechanisms may be vulnerable at the level where the virtualized code interacts with the underlying operating system. The resolution of whether such attacks can undermine security will help create better detection and analysis methods for malware that also employ software virtualization. Our findings help frame research for additional mitigation techniques using hardware-based integration or hybrid virtualization techniques that can better defend legitimate uses of virtualized software protection.