In recent years, High Performance Computing (HPC) systems have been used for scientific data analyses and simulations defined as workflows, leading to an increasing demand for efficient workflow execution on HPC systems. However, conventional HPC systems are optimized to execute a large number of independent jobs efficiently and face challenges such as excessive resource allocation and increased turnaround time when executing workflows consisting of multiple dependent tasks. To address this issue, this paper proposes a job scheduling method to reduce the turnaround time of workflows. Specifically, it separates the control of the job execution order from the control of the task execution order. Additionally, when controlling the task execution order, it assigns execution priorities while considering task dependencies. As a result, compared to conventional methods that do not control task execution order, the proposed method significantly reduces the turnaround time.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Workflow Batch Job Scheduling with Considering Task Dependencies

  • Kaito Yanai,
  • Keichi Takahashi,
  • Yoichi Shimomura,
  • Hiroyuki Takizawa

摘要

In recent years, High Performance Computing (HPC) systems have been used for scientific data analyses and simulations defined as workflows, leading to an increasing demand for efficient workflow execution on HPC systems. However, conventional HPC systems are optimized to execute a large number of independent jobs efficiently and face challenges such as excessive resource allocation and increased turnaround time when executing workflows consisting of multiple dependent tasks. To address this issue, this paper proposes a job scheduling method to reduce the turnaround time of workflows. Specifically, it separates the control of the job execution order from the control of the task execution order. Additionally, when controlling the task execution order, it assigns execution priorities while considering task dependencies. As a result, compared to conventional methods that do not control task execution order, the proposed method significantly reduces the turnaround time.