Protocol reverse engineering plays a critical role for Vulnerability Discovery and Security Analysis. Protocol reverse engineering tools based on network traffic analyze captured communication data to infer protocol formats and semantics. However, these methods globally cluster messages and analyze each cluster separately, which causes the loss of valuable field information. To address this problem, we present \(\textbf{MDIplier}^*\) , a protocol reverse engineering tool that leverages the hierarchical structure of protocol messages and performs customized analysis at each message layer. \(\textbf{MDIplier}^*\) performs an iterative inference process. During each iteration, it identifies the message delimiter with our algorithm designed for layer separation and infers the format for each layer separately, optimizing the use of available field information. Meanwhile, we optimize the alignment algorithm and solve the question that MDIplier cannot analyze large-scale messages. Our evaluation of 11 widely used protocols shows that \(\textbf{MDIplier}^*\) outperforms state-of-the-art methods. After optimization, its field inference achieves a perfection score 1.75 \(\times\) , 2.3 \(\times\) , 1.8 \(\times\) , 7 \(\times\) , and 3.5 \(\times\) higher than those of Netplier, BinaryInferno, Netzob, FieldHunter, and Nemesys, respectively. Experimental analysis across protocols with diverse features demonstrates the feasibility of hierarchical format analysis and validates the effectiveness of our algorithm. Furthermore, the experiments on proprietary protocols used in three IoT devices demonstrate the effectiveness of \(\textbf{MDIplier}^*\) in real-world scenarios.
