Scholarlink

<p>Large-scale research infrastructures increasingly depend on distributed computing platforms to deliver timely data processing and analysis to their scientific communities. In such environments, job turnaround time is a key operational quantity because it affects user-perceived latency, workflow planning, and the efficient use of heterogeneous computing resources across facilities processing millions of jobs per week. Predicting turnaround time is difficult because it depends not only on intrinsic job characteristics but also on dynamic infrastructure conditions such as queue pressure, resource availability, brokerage decisions, and site-specific operating behavior. We study this problem in the Production and Distributed Analysis (PanDA) workload management system, which supports data-intensive scientific computing across grid, cloud, and high-performance computing resources and is used by large scientific collaborations including ATLAS at the Large Hadron Collider and ePIC at the Electron-Ion Collider. We present PanDA-GNN, a graph neural network for job-level turnaround-time prediction that represents jobs together with their local execution context. Using data from five of the most active PanDA computing sites, with approximately 2.1 million training jobs and separate validation and test sets of approximately 0.46 million and 0.40 million jobs, respectively, we show that graph-based context improves predictive performance over strong non-graph baselines. On the held-out test set, PanDA-GNN achieved R<sup>2</sup> = 0.94, MAE = 84.72 minutes, MedAE = 39.62 minutes, RMSE = 163.59 minutes, and MAPE = 20.39%. These results show that graph-based workload modeling is a practical approach for turnaround-time prediction in large-scale research computing infrastructures and, more broadly, illustrate how data-driven methods can support informed scheduling, provisioning, and latency-aware workflow management across shared distributed research computing infrastructures.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Predicting Job Turnaround Time in Large-Scale Distributed Computing Environments with Graph Neural Networks

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月27日

查看全文
  • Tasnuva Chowdhury,
  • Tadashi Maeno,
  • Fatih Furkan Akman,
  • Joseph Boudreau,
  • Sankha Dutta,
  • Shengyu Feng,
  • Adolfy Hoisie,
  • Kuan-Chieh Hsu,
  • Raees Khan,
  • Jaehyung Kim,
  • Ozgur O. Kilic,
  • Scott Klasky,
  • Tatiana Korchuganova,
  • Kenny Lo,
  • Verena Ingrid Martinez Outschoorn,
  • Paul Nilsson,
  • David K. Park,
  • Norbert Podhorszki,
  • Yihui Ren,
  • John Rembrandt Steele,
  • Frédéric Suter,
  • Sairam Sri Vatsavai,
  • Wei Yang,
  • Yiming Yang,
  • Shinjae Yoo,
  • Alexei Klimentov

摘要

Large-scale research infrastructures increasingly depend on distributed computing platforms to deliver timely data processing and analysis to their scientific communities. In such environments, job turnaround time is a key operational quantity because it affects user-perceived latency, workflow planning, and the efficient use of heterogeneous computing resources across facilities processing millions of jobs per week. Predicting turnaround time is difficult because it depends not only on intrinsic job characteristics but also on dynamic infrastructure conditions such as queue pressure, resource availability, brokerage decisions, and site-specific operating behavior. We study this problem in the Production and Distributed Analysis (PanDA) workload management system, which supports data-intensive scientific computing across grid, cloud, and high-performance computing resources and is used by large scientific collaborations including ATLAS at the Large Hadron Collider and ePIC at the Electron-Ion Collider. We present PanDA-GNN, a graph neural network for job-level turnaround-time prediction that represents jobs together with their local execution context. Using data from five of the most active PanDA computing sites, with approximately 2.1 million training jobs and separate validation and test sets of approximately 0.46 million and 0.40 million jobs, respectively, we show that graph-based context improves predictive performance over strong non-graph baselines. On the held-out test set, PanDA-GNN achieved R2 = 0.94, MAE = 84.72 minutes, MedAE = 39.62 minutes, RMSE = 163.59 minutes, and MAPE = 20.39%. These results show that graph-based workload modeling is a practical approach for turnaround-time prediction in large-scale research computing infrastructures and, more broadly, illustrate how data-driven methods can support informed scheduling, provisioning, and latency-aware workflow management across shared distributed research computing infrastructures.