High-performance computing (HPC) is undergoing a significant transformation as workloads become increasingly complex. The proliferation of nested parallelism and collocated functionalities in scientific software has naturally increased software complexity, leading to challenges in expressing launch configurations in a portable manner. This paper proposes the concept of self-unfolding dynamic workflows, which aims to alleviate this bottleneck by defining a compact runtime and mapping syntax that enables compute locality and resource composition between multiple jobs. Our scale-agnostic syntax for resource composition allows for job expressivity and dynamic resource utilization, making it easier to deploy and dynamically manage resources for parallel applications. We present a prototype implementation of this syntax over Slurm and an online visualization tool, demonstrating the advantages of this approach in terms of job layout compactness and usability.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Towards a Scale Invariant Syntax for Dynamic Job-Level Workflows

  • Jean-Baptiste Besnard,
  • Martin Schreiber,
  • Allen D. Malony

摘要

High-performance computing (HPC) is undergoing a significant transformation as workloads become increasingly complex. The proliferation of nested parallelism and collocated functionalities in scientific software has naturally increased software complexity, leading to challenges in expressing launch configurations in a portable manner. This paper proposes the concept of self-unfolding dynamic workflows, which aims to alleviate this bottleneck by defining a compact runtime and mapping syntax that enables compute locality and resource composition between multiple jobs. Our scale-agnostic syntax for resource composition allows for job expressivity and dynamic resource utilization, making it easier to deploy and dynamically manage resources for parallel applications. We present a prototype implementation of this syntax over Slurm and an online visualization tool, demonstrating the advantages of this approach in terms of job layout compactness and usability.