High Performance Computing (HPC) systems currently rely on fixed user-provided estimates of job execution times. These estimates are often inaccurate, resulting in inefficient resource use and the loss of unsaved work if a job times out shortly before reaching its next checkpoint. This work proposes a novel feedback-driven autonomy loop that dynamically adjusts the HPC job time limits based on the checkpointing progress reported by the applications. Our approach monitors checkpointing intervals of currently running jobs and the estimated start times of queued jobs, enabling informed decisions to either early cancel a job after its last completed checkpoint or extend the time limit sufficiently to accommodate the next checkpoint. The objective is to minimize tail waste, that is, the computation that occurs between the last checkpoint and the termination of a job, which is not saved and hence wasted. Through experiments conducted on a subset of a production workload trace, we show a 95% reduction of tail waste, which equates to saving approximately 1.3% of the total CPU time (cores  \(\times \)  sec) that would otherwise be wasted. We propose various policies that combine early cancellation and time limit extension, achieving tail waste reduction while improving scheduling metrics such as weighted average job wait time. The proposed autonomy loop improves scheduling in HPC environments, where system job schedulers and applications collaborate to significantly reduce resource waste and improve scheduling performance.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

An Autonomy Loop for Dynamic HPC Job Time Limit Adjustment

  • Thomas Jakobsche,
  • Osman Seckin Simsek,
  • Jim Brandt,
  • Ann Gentile,
  • Florina M. Ciorba

摘要

High Performance Computing (HPC) systems currently rely on fixed user-provided estimates of job execution times. These estimates are often inaccurate, resulting in inefficient resource use and the loss of unsaved work if a job times out shortly before reaching its next checkpoint. This work proposes a novel feedback-driven autonomy loop that dynamically adjusts the HPC job time limits based on the checkpointing progress reported by the applications. Our approach monitors checkpointing intervals of currently running jobs and the estimated start times of queued jobs, enabling informed decisions to either early cancel a job after its last completed checkpoint or extend the time limit sufficiently to accommodate the next checkpoint. The objective is to minimize tail waste, that is, the computation that occurs between the last checkpoint and the termination of a job, which is not saved and hence wasted. Through experiments conducted on a subset of a production workload trace, we show a 95% reduction of tail waste, which equates to saving approximately 1.3% of the total CPU time (cores  \(\times \)  sec) that would otherwise be wasted. We propose various policies that combine early cancellation and time limit extension, achieving tail waste reduction while improving scheduling metrics such as weighted average job wait time. The proposed autonomy loop improves scheduling in HPC environments, where system job schedulers and applications collaborate to significantly reduce resource waste and improve scheduling performance.