<p>Knowledge distillation (KD) transfers knowledge from a large teacher network to a lightweight student network, primarily through logit or feature distillation. While logit and feature distillation have been extensively studied, they face efficiency challenges in real-world applications, particularly when integrating logits with feature learning. These methods often fail to distinguish critical from non-critical regions in feature maps, leading to information loss and reduced efficiency. To overcome these issues, we propose a multilevel entropy decoupling and class-adaptive attention distillation framework. It dynamically adjusts entropy to regulate information flow, enabling precise knowledge transfer across instance, batch, and class levels. Additionally, we introduce a class-adaptive attention mechanism that simulates the teacher network’s analytical strategies, improving the student network’s adaptability and generalization in complex data environments. The proposed approach outperforms state-of-the-art distillation methods on ResNet, VGG, and WRN models using CIFAR-100 and ImageNet-1K datasets.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Class-adaptive attention transfer and multilevel entropy decoupled knowledge distillation

  • Xincai Lu,
  • Zhanquan Sun,
  • Chenjie Zou,
  • Chun He,
  • Xinping Hu

摘要

Knowledge distillation (KD) transfers knowledge from a large teacher network to a lightweight student network, primarily through logit or feature distillation. While logit and feature distillation have been extensively studied, they face efficiency challenges in real-world applications, particularly when integrating logits with feature learning. These methods often fail to distinguish critical from non-critical regions in feature maps, leading to information loss and reduced efficiency. To overcome these issues, we propose a multilevel entropy decoupling and class-adaptive attention distillation framework. It dynamically adjusts entropy to regulate information flow, enabling precise knowledge transfer across instance, batch, and class levels. Additionally, we introduce a class-adaptive attention mechanism that simulates the teacher network’s analytical strategies, improving the student network’s adaptability and generalization in complex data environments. The proposed approach outperforms state-of-the-art distillation methods on ResNet, VGG, and WRN models using CIFAR-100 and ImageNet-1K datasets.