Protocol syntax recovery via knowledge transfer

Abstract

Protocol reverse engineering plays a critical role in many security applications. However, current technologies rely primarily on network trace analysis, which often has syntax-inference limitations. Meanwhile, protocols regularly evolve to improve functionality and address vulnerabilities, adapting to changing needs and technological advances. This evolution has accumulated a wealth of prior knowledge for protocol reverse engineering, but it remains largely untapped and unused.

This paper presents SynRe, a protocol syntax reverse engineering method designed to address the challenge of recovering unknown message formats. SynRe exploits the wealth of prior syntax knowledge accumulated through protocol evolution and the inherent structural change characteristics of binary sequences. The approach first uses a language representation model that incorporates natural language knowledge and prior syntax knowledge of known messages to learn the mapping relationship between protocol semantics and syntax. The method then extracts the inherent structural change characteristics of binary message sequences using bit operations, facilitating the recovery of message syntax.

Our evaluation on five widely used protocol families shows that SynRe achieves encouraging score, significantly outperforming the state-of-the-art methods like Netzob+, Netzob, BinaryInferno, Netplier, and FieldHunter. Furthermore, SynRe achieves higher perfection than the trivial application of the original grammar for the protocols before evolution, demonstrating the effectiveness of knowledge transfer. Meanwhile, the experiments of adapting SynRe on protocol message with different diversity and sizes demonstrate that SynRe is not significantly affected by the size or diversity of the datasets involved.