Performance Optimization of HPC Workloads in Cloud Using AI-Driven Algorithms
摘要
As High-Performance Computing (HPC) workloads increasingly migrate to cloud infrastructures, the need for intelligent, inference-time adaptive scheduling becomes critical. Conventional schedulers such as FCFS or SJF often struggle to adapt to the heterogeneous and dynamic nature of cloud-based systems, leading to inefficient resource utilization and increased job wait times. This paper proposes a multi-stage AI-based pipeline to address these challenges through the integration of three core capabilities: job runtime prediction using supervised learning, anomaly detection via deep autoencoders, and adaptive resource scheduling using reinforcement learning. Leveraging real-world data from the MIT SuperCloud dataset, our system extracts meaningful patterns from time-series telemetry to support informed scheduling decisions. The job prediction module estimates runtimes based on CPU utilization, memory consumption, and I/O statistics. The anomaly detection module flags abnormal jobs using learned GPU performance norms. The RL scheduler dynamically matches jobs to compute nodes based on predicted duration and anomaly status, optimizing for turnaround time and utilization. Experimental evaluations demonstrate a 28% reduction in average turnaround time and over 10% increase in resource utilization compared to traditional schedulers. These results establish the viability of AI-driven orchestration strategies in HPC cloud platforms and underscore the importance of integrated learning-based systems in achieving scalable, efficient, and context-aware workload management.