A Method for Combining Heterogeneous Workflows in HPC Systems

HPC job scheduling relies on user-provided parameters, including the requested resources (number of computing nodes or cores) and the requested time (user-estimated runtime). These parameters are set by most schedulers when queueing a job and remain unchanged throughout the job lifecycle. Difference in time and resources requests for jobs leads to windows (gaps) with idle resources in the schedule. Such windows cause supercomputer nodes downtime. To reduce idle time, most schedulers use backfilling strategies. However, due to inaccurate user estimates of job runtime, windows show dynamic and stochastic behavior, reducing backfilling effectiveness. This paper proposes a post-scheduling method based on the use of an additional workflow of adaptive jobs. These jobs can adapt time and resources requirements according to the size of the available window. We present the simulation results, and evaluate the impact of post-scheduling on supercomputer resources utilization and job average bounded slowdown.