Strengthening attention: knowledge distillation via cross-layer feature fusion for image classification

Deep learning has achieved great success in computer vision, especially in image classification tasks. How to improve the generalization ability and compactness of deep neural networks has gradually attracted widespread attention from researchers. Knowledge distillation is an effective technique for model compression. It transfers general knowledge from a sophisticated teacher model to a smaller student model. Recently, some studies refine knowledge from feature maps or adopt complex attention mechanisms to better supervise students imitating teachers. However, their methods focus too much on how to improve students’ accuracy and largely overlook the associated training costs, which violates the original intention of knowledge distillation to compress the model. To achieve a balance between performance and efficiency, in this paper, we introduce a straightforward and effective distillation method to utilize the deepest feature maps to enhance shallow features. Specifically, our method performs processing only on the original feature maps without an extra assisting network. Moreover, we use cross-layer feature fusion to enhance the attention on shallow feature maps. By visualizing the features of different layers, we demonstrate the importance of the fusion operation in our method. Our experimental results on the CIFAR-100, tinyImageNet and miniImageNet datasets show that our approach outperforms previous methods, especially in the balance between performance and training cost. Further ablative studies verify the effectiveness of the design.