AGLFuzz: Automata-Guided Fuzzing for detecting logic errors in security protocol implementations

Abstract

Security protocols are crucial for ensuring communication security and safeguarding data integrity in computer networks and distributed systems. The complexity of security protocol logic, coupled with implementation challenges, often results in protocol implementations failing to satisfy the security requirements due to logical errors. Unlike memory-related bugs, logical errors do not exhibit fixed patterns or behaviors, thereby rendering them especially challenging to detect. Therefore, we propose a logic error detection method based on blackbox fuzzing. This method takes protocol interaction behavior as atomic proposition, utilizes linear temporal logic on finite traces ($LT{L}_f$) to express expected properties. Logical errors are identified according to whether the abstract interaction sequence extracted from the fuzz data can be accepted by the automata corresponding to the $LT{L}_f$ property. Furthermore, we design an automata-guided fuzz testing algorithm that leverages the state information of automatas to drive test sequence generation, thereby accelerating the error search process. To support this method, a general-purpose black-box fuzz testing framework, AGLFuzz, has been implemented, currently including testing modules for the TLS1.3 and IPsec protocol implementations. Experimental evaluations on several widely used TLS1.3 and IPsec protocol implementations have led to the discovery of multiple counterexamples that violated specific properties and vulnerabilities that could cause the target to crash. Notably, three of these vulnerabilities have been assigned CVE numbers, highlighting the effectiveness of the proposed method.