The management of complex computational workflows is a critical challenge across various domains, including bioinformatics, data science, and engineering. Traditional workflow management systems, such as Nextflow and Airflow, provide automation and task scheduling capabilities but often introduce significant complexity and overhead. In this paper, we introduce SDAG (Slurm Directed Acyclic Graph), a lightweight and efficient framework designed to automate workflows using the native job dependency management system of the SLURM scheduler. Unlike traditional WMS, SDAG eliminates the need for an external workflow controller by leveraging SLURM’s built-in features to handle job dependencies, ensuring tasks are executed in the correct order. We demonstrate the versatility and efficiency of SDAG by applying it to various use cases, including the computationally intensive ChIP-Seq workflow. In our experiments, we compare SDAG against Nextflow and Airflow in terms of execution time and scalability using datasets of varying sizes: small (2 GB), medium (10 GB), and large (50 GB). The results show that SDAG consistently outperforms both Nextflow and Airflow, particularly for large datasets, due to its minimal overhead and efficient use of parallelism within SLURM. This paper highlights the potential of SDAG as a general-purpose tool for automating workflows across different domains, offering a simple, scalable, and reliable solution for managing complex computational tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SDAG: A Directed Acyclic Graph Based Workflow Manager for SLURM

  • Abdulrahman Azab

摘要

The management of complex computational workflows is a critical challenge across various domains, including bioinformatics, data science, and engineering. Traditional workflow management systems, such as Nextflow and Airflow, provide automation and task scheduling capabilities but often introduce significant complexity and overhead. In this paper, we introduce SDAG (Slurm Directed Acyclic Graph), a lightweight and efficient framework designed to automate workflows using the native job dependency management system of the SLURM scheduler. Unlike traditional WMS, SDAG eliminates the need for an external workflow controller by leveraging SLURM’s built-in features to handle job dependencies, ensuring tasks are executed in the correct order. We demonstrate the versatility and efficiency of SDAG by applying it to various use cases, including the computationally intensive ChIP-Seq workflow. In our experiments, we compare SDAG against Nextflow and Airflow in terms of execution time and scalability using datasets of varying sizes: small (2 GB), medium (10 GB), and large (50 GB). The results show that SDAG consistently outperforms both Nextflow and Airflow, particularly for large datasets, due to its minimal overhead and efficient use of parallelism within SLURM. This paper highlights the potential of SDAG as a general-purpose tool for automating workflows across different domains, offering a simple, scalable, and reliable solution for managing complex computational tasks.