RECSRF: Novel Technique to Evaluate Program Security using Dynamic Disassembly of Machine Instructions

Abstract

Modern software systems have nearly an uncontrolled growth in complex requirements that is fueled by the need to interact with multiple other complex systems. While some of the complexity increases are due to activities such as bug fixes which are unavoidable, other activities such as expanding and enhancing functionality of a software system becomes a risky endeavor to undertake with respect to security as responding to the potential vulnerabilities due to complexity increases require scarce resources including technical expertise and time. This paper introduces RECSRF; a novel framework to quantitatively evaluate the security of an execution in line with the security risk impact it makes over the particular microprocessor on which it executes. RECSRF consists of two components; a novel concept called The Run-time Execution Complexity (REC) of a program execution, which evaluates the trade-off between performance vs. security, while adhering to Control Flow Integrity (CFI) of programs, and an information theoretic technique to approximate the Security Risk Factor (SRF), which approximates the risk of a particular execution by analyzing dynamically disassembled machine instructions of a particular microprocessor. The RECSRF value allows software designers to select the most secure resource combination among a given set of resources, and software implementers to decide whether to proceed or not with a software change. The method can also be used to detect control flow hijacks at runtime by using it as an application level intrusion detection mechanism which allows transforming the same to an application level intrusion preventer upon successful implementation.