ATMKD: adaptive temperature guided multi-teacher knowledge distillation

Knowledge distillation is a technique that aims to distill the knowledge from a large well-trained teacher model to a lightweight student model. In recent years, multi-teacher knowledge distillation has received widespread attention, exploring diverse knowledge sources from multiple teachers to provide students with more comprehensive guidance. However, existing multi-teacher distillation methods usually use a single aggregation strategy, ignoring the disparities among different types of knowledge. In addition, they usually set the temperature to a fixed value, ignoring the effect of temperature on multi-teacher knowledge distillation. To address these issues, we propose adaptive temperature guided multi-teacher knowledge distillation (ATMKD), which uses adaptive temperature and diverse aggregation strategy to improve distillation performance. Specifically, we internally leverage dynamic and learnable temperature to adaptively control the difficulty level of multi-teacher knowledge. Externally, the diverse aggregation strategy is used to fuse rich knowledge from multiple teachers. Optimizing the teacher output at both internal and external levels can provide more comprehensive guidance for the student model and achieve better distillation performance. The extensive experiments with various teacher and student architectures on multiple benchmark datasets show that the proposed approach outperforms other knowledge distillation methods. The code is available at https://github.com/JSJ515-Group/ATMKD.