HDEM: A Hierarchical Dynamic Ensemble Model for Accurate Runtime Prediction on High Performance Computing Systems
摘要
Accurate job runtime prediction is essential for efficient job scheduling, which in turn significantly contributes to system performance optimization in High Performance Computing (HPC) environments. However, the dynamic nature of HPC workloads, including fluctuations in job size, priority, and system conditions, poses considerable challenges for single prediction models. To address these challenges, this study proposes a novel Hierarchical Dynamic Ensemble Model (HDEM), featuring an ensemble-of-ensembles architecture that adapts to workload variations. HDEM integrates a prediction degradation detection mechanism to dynamically adjust the contributions of base learners. Evaluated on historical workload data from production HPC systems, the proposed method demonstrates superior performance compared to conventional models and provides more reliable runtime estimates. This approach not only enhances prediction accuracy but also ensures adaptability to evolving workload patterns in HPC environments.