A Scalable Python Framework for Static Malware Classification Using Feature Optimization and Ensemble Learning
摘要
The article describes a framework for automated malware classification based on Python that incorporates ML algorithms to learn from files (Portable Executable) from the EMBER 2018 dataset. A complete pipeline with 616 extracted features, model training, optimization, and evaluation is coded in Python 3.8 and implemented LIEF 0.16.5 library to account for older malware, with multiple file implementations through the code practice. Four ML algorithms – Random Forest, XGBoost, LighGBM, and CatBoost, are trained on multiple training set sizes and optimized using Optuna hyperparameter tuning. Feature selection is explored using SelectKBest metric with multiple dimensions. A scanner in a command line interface is created, which gives even more offline malware detection. A variety of metrics, graphics, and deployment considerations are described. The proposed solution allows malware detection with different use cases in constrained conditions or legacy systems that must be highly interpretable and scalable. The conducted investigations demonstrate very promising results regarding the ability to automate malware classification.