A novel memory usage guided fuzzing based on particle swarm optimization

Fuzzing has become the focus of research in the field of software testing due to its advantages such as simple testing process, easy deployment, and easy reproduction of crashes. However, fuzzing also suffers from the disadvantages of poor test purpose and invalid generated seeds. To solve the above problems, researchers proposed the Memory Usage Guided Fuzzing (MUGF). To further optimize the performance of MUGF, this paper proposes a fuzzing method named Memory Usage Guided Fuzzing based on Particle Swarm Optimization (MUGF-PSO). MUGF-PSO will guide the selection of mutation operators for subsequent testing by learning the effectiveness of each mutation operators during previous testing. Specifically, MUGF-PSO regards each mutation operator as a particle in the particle swarm optimization algorithm, and the coverage change is regarded as an important factor to evaluate the seed’s local and global best position. In addition, the efficient selection probability distribution of mutation operator is constantly iteratively searched to make the MUGF tends to select the mutation operator that is more able to trigger new coverage. Furthermore, the MUGF-PSO is integrated into MemLock developed by MUGF, that is, MemLock-PSO is developed. We conduct a comparison experiment with 5 fuzzers (including AFL, MemLock, EcoFuzz, HavocMAB and Darwin), and the results show that MemLock-PSO is able to find more paths and crashes in 12 widely used program with different functions from 10 tools, with a significant gap of around 12 h. Meanwhile, the MemLock-PSO have a significant difference in the number of paths and crashes compared to other fuzzers with a good stability. Our work also proves the remarkable significance of applying swarm intelligence optimization algorithms in fuzzing in order to solve the problem of selection of mutation operators.