Knowledge Distillation with Information Compressed Representations

In the current situation of increasing model parameters, knowledge distillation technology has received much attention for its ability to efficiently compress the knowledge of large models while preserving their primary performance. However, in the real world, the structures of the teacher and student models are not always consistent, potentially hindering the full transfer of knowledge from teacher to student. Furthermore, student models often face difficulties in acquiring the high-dimensional knowledge of teacher models due to the reduction in learnable parameters. In this paper, we propose an Information Compressed Representations-based Knowledge Distillation (ICR-KD) approach. Specifically, ICR-KD introduces a span constraint mechanism to enforce similarity in hierarchical distances between teacher and student models. Furthermore, ICR-KD leverages autoencoders to guide the student model towards acquiring low-dimensional knowledge from the teacher model, thereby enhancing the interaction between student and teacher models and consequently augmenting the student’s generalization capabilities. Our approach has demonstrated significant performance on five datasets in the GLUE benchmark and performed well on the remaining datasets.