Revisiting workflow execution in HPC: a data-flow approach
摘要
Scientific workflows are essential to modern scientific computing, yet traditional execution approaches-based on control-flow paradigms and disk-based data transfers-struggle as data movement, rather than computation, emerges as the dominant performance bottleneck. These methods suffer from long latency due to centralized orchestration, sequential task triggering, and inefficient disk-mediated exchanges. We propose HPCFlow, a data-flow-oriented workflow framework designed for high-performance computing (HPC) environments. HPCFlow supports decentralized, input-driven execution. Functions are decomposed into computation and data transmission, enabling asynchronous data propagation and efficient overlap. HPCFlow incorporates context-aware data transfer strategies and alleviates small-file I/O inefficiencies through mini-batching. Additionally, HPCFlow implements an input synchronization mechanism to guarantee data completeness during parallel execution under elastic scaling conditions. Empirical results from a production HPC environment demonstrate that compared to a control-flow baseline, HPCFlow significantly reduces makespan and end-to-end latency, achieves efficient overlap, and alleviates pressure on network file systems, thereby validating its effectiveness for data-intensive scientific workflows.