Class-adaptive attention transfer and multilevel entropy decoupled knowledge distillation
摘要
Knowledge distillation (KD) transfers knowledge from a large teacher network to a lightweight student network, primarily through logit or feature distillation. While logit and feature distillation have been extensively studied, they face efficiency challenges in real-world applications, particularly when integrating logits with feature learning. These methods often fail to distinguish critical from non-critical regions in feature maps, leading to information loss and reduced efficiency. To overcome these issues, we propose a multilevel entropy decoupling and class-adaptive attention distillation framework. It dynamically adjusts entropy to regulate information flow, enabling precise knowledge transfer across instance, batch, and class levels. Additionally, we introduce a class-adaptive attention mechanism that simulates the teacher network’s analytical strategies, improving the student network’s adaptability and generalization in complex data environments. The proposed approach outperforms state-of-the-art distillation methods on ResNet, VGG, and WRN models using CIFAR-100 and ImageNet-1K datasets.