A proposed approach to detect and thwart previously unknown code injection attacks

Abstract

This paper presents a proposed approach called VAIL System Call Monitor (YSCM) to detect and thwart previously unknown code injection attacks. The idea is based on the fact that any process needs to correctly invoke CreateProcessO system calls, otherwise child-process creation will fail. YSCM intercepts and verifies CreateProcessO system call invocations from a monitored process. In case an unknown executable is detected in the first parameter of a call, this indicates its maliciousness. In response, YSCM encrypts that parameter value to render the call invalid, thereby thwarting adversaries' attacks by preventing the operating system from loading and executing the new malicious child process. YSCM runs in a microkernel-based virtual machine in order to achieve two-fold advantages: (1) isolate security-critical information from probable adversaries' attacks; and (2) exploit security-related and performance-related advantages associated with thin virtual machine monitors. The expected effectiveness of YSCM is high since it is circumvention-proof, and precise in extracting the normal behavior of applications chosen to be monitored.