Layer-Wised Sparsification Based on Hypernetwork for Distributed NN Training

As Deep Neural Networks (DNNs) evolve in complexity, so does their parameter size, resulting in prolonged training time. While various distributed training strategies have been proposed to speed up training, the efficiency of these strategies is often hindered by the frequent communication required between different computational nodes. Numerous gradient compression techniques (e.g., Sparsification, Quantization, Low-Rank) have been introduced to enhance the communication process. However, these methods mainly focus on the numerical characteristics while neglecting the inherent characteristics of neural network training. In addition, these techniques necessitate the operation and transmission of tensors across all layers, which depletes the computational resources and requires substantial transmission time. To address these issues, this paper proposes a Layer-wised Sparsification method. Instead of compressing the gradients of all layers, the layers carefully selected through a Hypernetwork within each computational node will be transmitted. An efficient objective function is constructed for the Hypernetwork to guide the selection of layers for transmission, which ensures that layers that contribute more to the learning process are prioritized for transmission. Comprehensive experiments on Resnet-18 and VGG-16 are conducted to verify our method. The results show that the proposed method can reduce the communication overhead with only a slight loss of accuracy. Furthermore, our method can be combined with other compression methods, leading to further reductions in communication volume.