JAFC: Job-Aware Flow Control for Distributed DNN Training
摘要
The growing scale of deep neural network (DNN) has made distributed training essential, but concurrent jobs in shared clusters face serious network contention that limits overall performance. Existing centralized or host-side flow control approaches either face scalability bottlenecks or lack global awareness. To address these limitations, we propose JAFC, a job-aware flow control mechanism designed for distributed DNN training. JAFC leverages switches to support global state sharing across the cluster. Each work node periodically broadcasts its remaining communication volume and senses the progress of other nodes in a distributed manner. Based on this global view, each node adaptively adjusts its sending rate according to its relative remaining volume. This coordination helps stagger communication phases across jobs, thereby reducing communication overlap and alleviating network congestion. By simultaneously adjusting the transmission rates of individual work nodes within a job and coordinating bandwidth allocation across multiple jobs, JAFC reduces the overall completion time of the training process. Simulation results indicate that compared to the state-of-the-art flow control mechanisms, the proposed mechanism reduces the communication time by 1.53% to 34.5% for DNN training of varying scales.