JREP - A Job Runtime Ensemble Predictor for Improving Scheduling Performance on High Performance Computing Systems

Efficient resource utilization in High Performance Computing (HPC) systems heavily relies on accurate job runtime prediction. This paper introduces JREP (Job Runtime Ensemble Predictor), a novel ensemble learning approach combining multiple prediction techniques to enhance HPC job runtime estimation accuracy. We evaluate JREP using real-world datasets from production HPC systems and integrate it with deviation backfilling, a prediction-aware scheduling method. Our results demonstrate that JREP not only outperforms individual prediction methods in accuracy but also significantly improves scheduling performance. This work contributes to optimizing HPC operations through advanced machine learning, offering a promising direction for enhancing overall system efficiency in diverse and dynamic HPC environments.