Adaptive mutation based on multi-population evolution strategy for greybox fuzzing

Abstract

The development of adaptive mutation techniques to enhance gray-box fuzzing performance has become trendy. However, existing adaptive methods have limitations in that they either ignore the impacts of different characteristics of seed inputs or require assumptions about the probability distribution model. Motivated by the observation, we present a novel adaptive mutation approach that combines seed clustering and Evolution Strategy to automatically find the optimal mutation scheduling method for seeds with different characteristics. Our approach captures seed inputs' structural and functional similarities and partitions them into proper populations. The Evolution Strategy is then used to iteratively optimize the probability distribution of operator selection for each population. We implement the prototype tool MesFuzz based on the aforementioned ideas. Evaluation on LAVA-M shows that MesFuzz is the only fuzzer to find bugs in all target programs. In addition, MesFuzz improves the path coverage by 132%, 14%, and 12% over DARWIN, SeamFuzz, and AFL++, respectively. That will facilitate fuzzing to discover vulnerabilities in real-world software and firmware further.