Real-Life HPC Workload Trace Featuring Refined Job Runtime Estimates
摘要
User-provided job walltime (runtime) estimates are often used in current job schedulers to optimize performance and predictability when scheduling parallel jobs on computing resources. Since the walltime estimate is usually the upper runtime limit after which a job is forcibly terminated, user-provided estimates are usually overestimated, inaccurate, and rather coarse-grained. In this situation, the performance of crucial scheduling policies (e.g., the ability to backfill jobs) can be degraded. In this paper, we present a large real-life workload trace from the Czech National Grid Infrastructure MetaCentrum that contains automatically generated runtime estimates that improve original (user-provided) job walltime estimates. We also present our experience with using an automated system to improve user-provided estimates in an actual HPC system. Last but not least, we provide insights into the nature of the workload and also share several tools used to develop, debug, and analyze our predictor, scheduling policies, and the workload itself.