Evaluating Optimal Phase Ordering in Obfuscation Executives

Obfuscation is a software protection technique that aims to increase the difficulty and amount of resources required to understand programs from the perspective of a malicious end user. The order and number of obfuscating transformations is determined by an obfuscation executive and the optimal arrangement of transformation defines the phase ordering problem. In this paper, we report on a case study evaluation for determining the optimal phase ordering for an obfuscation executive. We analyze obfuscation effectiveness of variants generated by Tigress, a dynamic virtualizing obfuscator with four transformation types. We test the evaluation of multiple orderings against a symbolic virtual machine to determine the strengths and weaknesses of each combination. We use overhead (cost) and effectiveness as the tradeoff space to determine the best sequence and ordering of transformations within this context. Our results show that, ideally, applying control flow transformation, data encoding, abstract transforms, and then dynamic virtualization provides the highest effectiveness on average against symbolic execution attacks.