Multi-Teacher Knowledge Distillation via Student’s Reflection
摘要
Multi-teacher knowledge distillation utilizes diverse information from multiple teacher models to guide the student model’s training. Traditional methods of knowledge distillation in multi-teacher scenarios typically follow a one-way approach, where teachers pass on their knowledge, compelling the student to replicate their output predictions. However, the student’s reflective process is an essential element of human learning in real-world scenarios. In this paper, we propose multi-teacher knowledge distillation with student’s reflection (MKD-SD), which integrates self-distillation to promote the student’s reflective learning process. Additionally, to further enhance the student network’s performance, we apply a decoupled knowledge distillation technique, which breaks down the standard loss function into two independent components. To maintain stability during training, we use intermediate layer features as the transferred knowledge. The effectiveness of MKD-SD was rigorously assessed using the CIFAR-100 benchmark. For example, the ShuffleNetV2 model trained with MKD-SD achieved a top-1 accuracy of 77.86% when a pretrained WideResNet40–2 model was used as the teacher, marking an absolute improvement of 4.97% compared to the model trained independently. Moreover, it consistently surpasses all existing state-of-the-art methods across different teacher-student model combinations.