As task complexity increases, ensuring the reliability of big data task scheduling systems becomes increasingly critical, yet achieving high reliability remains challenging. Robust anomaly detection combined with effective root cause localization is essential. Current methodologies face two major challenges. First, although these systems generate diverse data types—traces, system logs, and Key Performance Indicators (KPIs)—most approaches remain trace-centric, which can miss a holistic system view and overlook certain anomalies. Second, troubleshooting typically proceeds in two phases—anomaly detection and root cause localization—that are often treated independently, neglecting their interdependencies; errors in detection then propagate and impair localization. To address these issues, we propose DAGLoc, an end-to-end framework that jointly performs anomaly detection and root cause localization for big data task scheduling systems. Leveraging Graph Transformers, DAGLoc unifies both processes to enable more comprehensive and accurate troubleshooting. Experiments on multiple datasets show that DAGLoc consistently delivers strong performance, improving reliability and efficiency in resolving complex issues.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DAGLoc: End-to-End Troubleshooting Approach for Big Data Scheduling System

  • Xueyong Tan,
  • Shipeng Zhang,
  • Jing Liu

摘要

As task complexity increases, ensuring the reliability of big data task scheduling systems becomes increasingly critical, yet achieving high reliability remains challenging. Robust anomaly detection combined with effective root cause localization is essential. Current methodologies face two major challenges. First, although these systems generate diverse data types—traces, system logs, and Key Performance Indicators (KPIs)—most approaches remain trace-centric, which can miss a holistic system view and overlook certain anomalies. Second, troubleshooting typically proceeds in two phases—anomaly detection and root cause localization—that are often treated independently, neglecting their interdependencies; errors in detection then propagate and impair localization. To address these issues, we propose DAGLoc, an end-to-end framework that jointly performs anomaly detection and root cause localization for big data task scheduling systems. Leveraging Graph Transformers, DAGLoc unifies both processes to enable more comprehensive and accurate troubleshooting. Experiments on multiple datasets show that DAGLoc consistently delivers strong performance, improving reliability and efficiency in resolving complex issues.