<p>This survey provides a comprehensive overview of applying machine learning (ML) to high-performance computing (HPC) scheduling, addressing the growing demand for efficient resource management in complex scientific and engineering computations. We examined traditional HPC scheduling methods and identified their limitations in meeting contemporary challenges. The survey then systematically explores the integration of ML techniques, including supervised learning, unsupervised learning, reinforcement learning, and neural networks, as well as evolutionary algorithms across various aspects of HPC scheduling. These aspects encompass job allocation, resource management, workload prediction, task mapping, load balancing, performance optimization, fault tolerance, energy efficiency, and security compliance. A comparative analysis of recent studies reveals that these ML approaches have demonstrated significant improvements in key performance metrics, with reductions in job completion time of up to 69%, increases in resource utilization of up to 24%, and overall performance improvements of up to 65%. These findings highlight the potential of ML methods to drastically improve HPC resource utilization and job throughput compared to traditional scheduling methods. By synthesizing recent advancements, providing a critical assessment of ML’s potential in HPC scheduling, and offering a quantitative comparison of various ML techniques, this survey serves as a valuable resource for researchers, practitioners, and decision makers in the field of ML-based HPC scheduling optimization.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Machine learning approaches for optimizing high-performance computing scheduling: a comprehensive survey and analysis

  • Kyrian Adimora,
  • Sai Rithvik Gundla,
  • Hongyang Sun

摘要

This survey provides a comprehensive overview of applying machine learning (ML) to high-performance computing (HPC) scheduling, addressing the growing demand for efficient resource management in complex scientific and engineering computations. We examined traditional HPC scheduling methods and identified their limitations in meeting contemporary challenges. The survey then systematically explores the integration of ML techniques, including supervised learning, unsupervised learning, reinforcement learning, and neural networks, as well as evolutionary algorithms across various aspects of HPC scheduling. These aspects encompass job allocation, resource management, workload prediction, task mapping, load balancing, performance optimization, fault tolerance, energy efficiency, and security compliance. A comparative analysis of recent studies reveals that these ML approaches have demonstrated significant improvements in key performance metrics, with reductions in job completion time of up to 69%, increases in resource utilization of up to 24%, and overall performance improvements of up to 65%. These findings highlight the potential of ML methods to drastically improve HPC resource utilization and job throughput compared to traditional scheduling methods. By synthesizing recent advancements, providing a critical assessment of ML’s potential in HPC scheduling, and offering a quantitative comparison of various ML techniques, this survey serves as a valuable resource for researchers, practitioners, and decision makers in the field of ML-based HPC scheduling optimization.