Collaborative and Progressive Teacher-Assistant Knowledge Distillation
摘要
Knowledge distillation enables compact student models to learn from larger teachers, achieving competitive accuracy with fewer parameters. Despite its efficiency, challenges remain in effectively transferring knowledge when architectural gaps exist between teacher and student. To mitigate this, Teacher-Assistant Knowledge Distillation (TAKD) introduces an intermediate assistant model to bridge the gap. While later researches have improved assistant architectures and training objectives, existing methods still suffer from suboptimal classification guidance, as the student learns from an assistant classifier that approximates the original teacher’s classification capability and lacks of a progressive learning scheme that adapts to the student’s evolving training needs. In this paper, we propose Collaborative and Progressive Teacher-Assistant Knowledge Distillation (CPTAKD), which guides student learning by jointly aligning feature representations from both assistant and original teacher. Meanwhile, a fused classifier is directly reused which is constructed by integrating the pretrained classifiers of two teachers. Furthermore, we design a guidance schedule that gradually shifts supervision from the assistant to the original teacher, enabling a smooth transition from simple to complex knowledge. Extensive experiments prove the effectiveness of our proposed method, showing superior performance over related distillation works.