The growing scale of deep neural network (DNN) has made distributed training essential, but concurrent jobs in shared clusters face serious network contention that limits overall performance. Existing centralized or host-side flow control approaches either face scalability bottlenecks or lack global awareness. To address these limitations, we propose JAFC, a job-aware flow control mechanism designed for distributed DNN training. JAFC leverages switches to support global state sharing across the cluster. Each work node periodically broadcasts its remaining communication volume and senses the progress of other nodes in a distributed manner. Based on this global view, each node adaptively adjusts its sending rate according to its relative remaining volume. This coordination helps stagger communication phases across jobs, thereby reducing communication overlap and alleviating network congestion. By simultaneously adjusting the transmission rates of individual work nodes within a job and coordinating bandwidth allocation across multiple jobs, JAFC reduces the overall completion time of the training process. Simulation results indicate that compared to the state-of-the-art flow control mechanisms, the proposed mechanism reduces the communication time by 1.53% to 34.5% for DNN training of varying scales.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

JAFC: Job-Aware Flow Control for Distributed DNN Training

  • Siyan Pan,
  • Xiaobin Tan,
  • Tiance Li,
  • Yingying Zeng,
  • Ning Xu,
  • Feng Yang

摘要

The growing scale of deep neural network (DNN) has made distributed training essential, but concurrent jobs in shared clusters face serious network contention that limits overall performance. Existing centralized or host-side flow control approaches either face scalability bottlenecks or lack global awareness. To address these limitations, we propose JAFC, a job-aware flow control mechanism designed for distributed DNN training. JAFC leverages switches to support global state sharing across the cluster. Each work node periodically broadcasts its remaining communication volume and senses the progress of other nodes in a distributed manner. Based on this global view, each node adaptively adjusts its sending rate according to its relative remaining volume. This coordination helps stagger communication phases across jobs, thereby reducing communication overlap and alleviating network congestion. By simultaneously adjusting the transmission rates of individual work nodes within a job and coordinating bandwidth allocation across multiple jobs, JAFC reduces the overall completion time of the training process. Simulation results indicate that compared to the state-of-the-art flow control mechanisms, the proposed mechanism reduces the communication time by 1.53% to 34.5% for DNN training of varying scales.