Obfuscated Malware Classification Using Hierarchy-Based Pipeline

Obfuscated malware detection and classification is vital in ensuring overall network security. Traditionally, obfuscated malware is detected and classified using handcrafted rules. While some of these systems, such as YARA (Yet Another Recursive Acronym), exhibit excellent detection performance, they are computationally expensive, slow, and unable to detect unknown attacks. As attacks evolve rapidly, it is necessary to update these models frequently, which is unscalable as it is done manually. Machine learning (ML) techniques for obfuscated malware detection promise to solve these issues. Recently, a publicly available dataset, CIC-MalMem-2022, was published that may serve as a realistic benchmark for training and evaluating ML malware detection and classification models. This paper proposes a hierarchical pipeline of ML models for malware detection and classification trained on the CIC-MalMem-2022 dataset. It achieves a class average F1 score of 79.98%, which is ~ 5% lower than the current state-of-the-art solution based on stacking ensemble and ~7% lower than the solution based on auto-generated rules. However, adapting handcrafted rules to new attacks requires significant effort from domain experts, while our solution can automatically adapt. It has a high production robustness, as it is comprised of several smaller, easy-to-re-train models. Furthermore, our hierarchical pipeline improves informativity by providing confidence scores for each classification stage and has a higher inference speed. Our ML model can be used to enhance the current state-of-the-art rule-based system by improving its detection speed and supporting domain experts in creating detection rules for novel attack types.