Quantifying the Effectiveness of Software Diversity using Near-Duplicate Detection Algorithms

Abstract

Software diversity is touted as a way to substantially increase the cost of cyber attacks by limiting an attacker's ability to reuse exploits across diversified variants of an application. Despite the number of diversity techniques that have been described in the research literature, little is known about their effectiveness. In this paper, we consider near-duplicate detection algorithms as a way to measure the static aspects of software diversity---viz., their ability to recognize variants of an application. Due to the widely varying results reported by previous studies, we describe a novel technique for measuring the similarity of applications that share libraries. We use this technique to systematically compare various near-duplication detection algorithms and demonstrate their wide range in effectiveness, including for real-world tasks such as malware triage. In addition, we use these algorithms as a way to assess the relative strength of various diversity strategies, from recompilation with different compilers and optimization levels to techniques specifically designed to thwart exploit reuse. Our results indicate that even small changes to a binary disproportionately affect the similarity reported by near-duplicate detection algorithms. In addition, we observe a wide range in the effectiveness of various diversity strategies.