Knowledge distillation typically involves transferring knowledge from large-scale teacher models to student models, garnering widespread attention in the fields of model compression and knowledge transfer. However, existing research often encounters several challenges. Firstly, the transmission process usually fixes the teacher model, making it difficult to adjust the learning situation of the student model according to individual needs. Secondly, the difficulty of tasks affects the learning ability of the student model. In training tasks, difficulty levels are often coupled with data, posing a challenge in guiding the student model to gradually master knowledge. To address these issues, this study proposes a dynamic learning temperature-based meta-learning knowledge distillation method, namely Temperature Meta-learning Knowledge Distillation (TMKD). Inspired by meta-learning heuristics, this algorithm enables the teacher model to dynamically adjust knowledge transfer strategies based on student feedback, facilitating tailored teaching. Furthermore, a Dynamic Temperature Regulation Module (DTRM) is constructed to flexibly control the difficulty level of tasks, allowing the student model to progressively learn knowledge. Finally, we design a Selective Insight Attention mechanism to ensure that the student network focuses more on key information during learning and inference, thereby enhancing overall performance. Extensive experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of our method.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Step-by-Step and Tailored Teaching: Dynamic Knowledge Distillation

  • Zhenqiang Zhang,
  • Liting Geng,
  • Wenqing Du,
  • Feng Li,
  • Chunxiao Wang,
  • Zhigang Zhao

摘要

Knowledge distillation typically involves transferring knowledge from large-scale teacher models to student models, garnering widespread attention in the fields of model compression and knowledge transfer. However, existing research often encounters several challenges. Firstly, the transmission process usually fixes the teacher model, making it difficult to adjust the learning situation of the student model according to individual needs. Secondly, the difficulty of tasks affects the learning ability of the student model. In training tasks, difficulty levels are often coupled with data, posing a challenge in guiding the student model to gradually master knowledge. To address these issues, this study proposes a dynamic learning temperature-based meta-learning knowledge distillation method, namely Temperature Meta-learning Knowledge Distillation (TMKD). Inspired by meta-learning heuristics, this algorithm enables the teacher model to dynamically adjust knowledge transfer strategies based on student feedback, facilitating tailored teaching. Furthermore, a Dynamic Temperature Regulation Module (DTRM) is constructed to flexibly control the difficulty level of tasks, allowing the student model to progressively learn knowledge. Finally, we design a Selective Insight Attention mechanism to ensure that the student network focuses more on key information during learning and inference, thereby enhancing overall performance. Extensive experiments on CIFAR-100 and ImageNet demonstrate the effectiveness of our method.