Crafting Binary Protocol Reversing via Deep Learning With Knowledge-Driven Augmentation

Abstract

Protocol reverse engineering (PRE) serves as an instrumental tool in various security research, such as protocol fuzzing and intrusion detection. Its primary objective lies in uncovering the format, semantics, and behavior of an unknown protocol without prior information. This paper presents DL-ProS2, a deep learning-based approach for binary protocol reversing, focusing on format segmentation and semantic inference from network traffic. Our approach is underpinned by highlighting the effectiveness of multi-scale features within the network traffic for identifying various types of fields and semantics. Based on this, DL-ProS2 employs a comprehensive end-to-end model that integrates U-Net, siamese network, and BiLSTM-CRF, which enables the effective analysis of unknown protocol traffic to extract the field boundaries and semantics. Meanwhile, to address the issue of limited data diversity and coverage, we implement an innovative knowledge-driven traffic simulation technique. This method harnesses the ChatGPT to extract protocol knowledge from publicly available protocol documents, such as RFCs, as the foundational rules for the simulation. Empirical results substantiate the efficacy of our approach, demonstrating precision rates exceeding 0.95 and recall rates surpassing 0.97 for partially unknown protocol format segmentation and semantic inference. It also retains effectiveness in the inference of completely unknown protocols, with average precision and recall rates of 0.69 and 0.62 for format segmentation, and 0.43 and 0.47 for semantic inference, respectively.