Optimizing DAG Scheduling in Data Pipelines Using Reinforcement Learning
摘要
Data pipelines are widely used to handle large amounts of information in modern computing. These pipelines are often modeled as Directed Acyclic Graphs (DAGs), where each task depends on others before it can run. A key challenge is to schedule tasks on available resources in a way that avoids long waiting times, wasted resources, or unnecessary delays. Most existing scheduling techniques rely on fixed rules, which work only under stable conditions and fail when workloads or resources change. This creates serious performance issues in real-world distributed systems. The proposed work applies reinforcement learning (RL) to improve DAG scheduling in data pipelines. In this method, scheduling is treated as a learning process where an agent makes decisions and adapts based on past results. The agent learns which scheduling actions reduce waiting time, improve resource balance, and keep the pipeline running smoothly. Unlike traditional methods, this adaptive approach reacts to new patterns in task execution and changes in resource availability without needing manual adjustments. The test-bed results suggest that RL is a strong candidate for efficient and adaptive DAG scheduling in data-driven environments.