Job Grouping Based Intelligent Resource Prediction Framework
摘要
In High-Performance Computing (HPC) systems, users estimate the resources they need for job submissions based on their best knowledge. However, underestimating the required execution time, number of processors, or memory size can lead to early job terminations. On the other hand, overestimating resource requests leads to inefficiencies in job backfilling, wasted compute power, unused memory, and poor job scheduling, ultimately reducing the overall system efficiency. As we enter the exascale era, we want to utilize resources more efficiently than ever. Existing schedulers lack mechanisms that predict the resource requirements of batch jobs. To address this challenge, we design a data-driven recommendation framework that leverages historical job information to predict three key parameters for batch jobs: the execution time, the maximum memory size, and the maximum number of CPU cores required. In contrast to existing machine learning-based resource prediction methods, we introduce an online resource suggestion framework that considers both underestimates and overestimates in the batch jobs’ resource provisioning. Our framework outperforms the baseline method with no grouping mechanism by achieving over 98% success in eliminating underpredictions and reducing the amount of overpredictions.