<p>Scientific workflows are essential to modern scientific computing, yet traditional execution approaches-based on control-flow paradigms and disk-based data transfers-struggle as data movement, rather than computation, emerges as the dominant performance bottleneck. These methods suffer from long latency due to centralized orchestration, sequential task triggering, and inefficient disk-mediated exchanges. We propose HPCFlow, a data-flow-oriented workflow framework designed for high-performance computing (HPC) environments. HPCFlow supports decentralized, input-driven execution. Functions are decomposed into computation and data transmission, enabling asynchronous data propagation and efficient overlap. HPCFlow incorporates context-aware data transfer strategies and alleviates small-file I/O inefficiencies through mini-batching. Additionally, HPCFlow implements an input synchronization mechanism to guarantee data completeness during parallel execution under elastic scaling conditions. Empirical results from a production HPC environment demonstrate that compared to a control-flow baseline, HPCFlow significantly reduces makespan and end-to-end latency, achieves efficient overlap, and alleviates pressure on network file systems, thereby validating its effectiveness for data-intensive scientific workflows.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Revisiting workflow execution in HPC: a data-flow approach

  • Tao Chen,
  • Xiaoning Wang,
  • Guanlong Li,
  • Yining Zhao,
  • Haili Xiao

摘要

Scientific workflows are essential to modern scientific computing, yet traditional execution approaches-based on control-flow paradigms and disk-based data transfers-struggle as data movement, rather than computation, emerges as the dominant performance bottleneck. These methods suffer from long latency due to centralized orchestration, sequential task triggering, and inefficient disk-mediated exchanges. We propose HPCFlow, a data-flow-oriented workflow framework designed for high-performance computing (HPC) environments. HPCFlow supports decentralized, input-driven execution. Functions are decomposed into computation and data transmission, enabling asynchronous data propagation and efficient overlap. HPCFlow incorporates context-aware data transfer strategies and alleviates small-file I/O inefficiencies through mini-batching. Additionally, HPCFlow implements an input synchronization mechanism to guarantee data completeness during parallel execution under elastic scaling conditions. Empirical results from a production HPC environment demonstrate that compared to a control-flow baseline, HPCFlow significantly reduces makespan and end-to-end latency, achieves efficient overlap, and alleviates pressure on network file systems, thereby validating its effectiveness for data-intensive scientific workflows.