Adaptive Gradient Data Partition and Route Selection for Distributed DNN Training
摘要
As the scale of distributed training for Deep Neural Network (DNN) increases, communication has become a critical performance bottleneck in data center networks. In-Network Aggregation (INA) can accelerate aggregating gradients process, the critical communication bottleneck in distributed DNN training. Due to the limited processing capacity of network switch, existing aggregation approaches have turned to a combination of INA and Parameter Server (PS) methods. However, these approaches employ a fixed partitioning of the gradient data, leading to suboptimal performance. This paper proposes GPRS, an adaptive gradient aggregation framework for distributed DNN training, which adaptively combines the line-rate processing of INA and large memory capacity of PS. We present an adaptive gradient data partition scheme, which divides the gradient data into INA and PS aggregation. We propose a heuristic algorithm that determines the aggregation route selection by assigning priority to the available network paths. To validate our approach, we build a real-world experimental testbed and implement the proposed framework. Experimental results show that it can overcome the limitations of traditional INA-only methods and reduce communication time by 15% \(\sim \) 52% for DNN training of varying scales.