Feature Reconstruction Distillation with Self-attention

A recently proposed knowledge distillation method based on feature map transfer verifies that the intermediate layers of the teacher model can be used as effective targets for training the student model for better generalization. Existing researches mainly focus on how to efficiently transfer knowledge between the intermediate layers of teacher and student models. However, they ignore the increase in the number of channels in the intermediate layer, which will introduce redundant feature information, and there is also a lack of interaction between shallow features and deep features. To alleviate these two problems, we propose a new knowledge distillation method called Feature Reconstruction Knowledge Distillation (FRD). By reconstructing the features of the intermediate layer of the teacher model, the student model can learn more accurate feature information. In addition, to tackle the problem that feature maps have only local information, we use a self-attention mechanism to fuse shallow and deep features, thereby enhancing the model’s ability to handle global details. Through extensive experiments on different network architectures involving various teacher and student models, we observe that the proposed method significantly improves performance.