Correlation Guided Multi-teacher Knowledge Distillation

Knowledge distillation is a model compression technique that transfers knowledge from a redundant and strong network (teacher) to a lightweight network (student). Due to the limitations of a single teacher’s perspective, researchers advocate for the inclusion of multiple teachers to facilitate a more diverse and accurate acquisition of knowledge. However, the current multi-teacher knowledge distillation methods only consider the integrity of integrated knowledge from the teachers’ level in teacher weight assignments, which largely ignores the student’s preference for knowledge. This will result in inefficient and redundant knowledge transfer, thereby limiting the learning effect of the student network. To more efficiently integrate teacher knowledge suitable for student learning, we propose Correlation Guided Multi-Teacher Knowledge Distillation (CG-MTKD), which utilizes the feedback of the student’s learning effects to achieve the purpose of integrating the student’s preferred knowledge. Through extensive experiments on two public datasets, CIFAR-10 and CIFAR-100, we demonstrate that our method, CG-MTKD, can effectively integrate the knowledge of student preferences during teacher weight assignments.