Real Environment Tolerance Curve (RETC): Assessing Machine Learning Model Performance Degradation Under Extreme Class Imbalance
摘要
Evaluating machine learning models in cybersecurity often involves datasets with an equal ratio of benign and malicious samples. Previous studies employing conventional evaluation methods have assessed performance at a single, fixed class ratio. However, this evaluation approach has a critical flaw. It fails to show how its performance degrades when the class imbalance becomes severe. This issue causes a mismatch when deploying the model in a real environment. Therefore, we propose a new evaluation methodology: the Real Environment Tolerance Curve (RETC). RETC is a systematic method to assess model robustness under varying degrees of class imbalance. It involves generating test datasets with an increasing proportion of benign data and plotting key performance metrics against the imbalance ratio. The resulting visualization, the RETC plot, enables a quantitative evaluation of the ability of a model to maintain performance in benign-dominated environments. In this study, we first establish the theoretical foundation of the RETC method through mathematical analysis, clarifying how each metric behaves as class ratios change. Next, empirical experiments on network traffic and URL datasets demonstrate that RETC reveals critical performance differences missed by conventional single-ratio evaluations. Experiments with URL datasets revealed a model whose false positive rate surged by more than three times (2.5% to 8.7%) as the imbalance ratio increased from 1 to 10. This capability helps diagnose whether performance degradation is an apparent effect of metric properties or an essential weakness in model generalization. In conclusion, the RETC is a reliable diagnostic tool for predicting performance in real operations.