Scholarlink

<p>Previous knowledge distillation research has primarily focused on transferring information from the teacher network’s final layer, often neglecting the valuable guidance embedded within the intermediate layers. This oversight can lead to a lack of direction for the student model in focusing on crucial knowledge aspects. Therefore, this paper investigates methods for effectively extracting guidance information from the teacher network’s intermediate layers and proposes the <i>method of extracting important information from feature map for knowledge distillation</i> <i>(EIKD)</i>. EIKD utilizes a rank-based pruning method to extract matching guidance information from the teacher network’s intermediate layer feature map based on the size of the student’s feature map. This approach addresses the challenge of knowledge transfer caused by mismatched feature map sizes between the teacher and student models. Furthermore, to prevent the loss of correlation information between the teacher’s feature map channels during the pruning process, EIKD establishes a correlation information matrix for the extracted feature maps and guides the student to fit the relationships between the teacher’s intermediate feature map channels. This ensures that the student effectively receives both the guidance information and the feature map correlation information from the teacher’s intermediate layers. In image classification tasks, using ResNet-34 as the teacher, our method improves the ImageNet Top-1 accuracy of ResNet18 by an average of 2.14%. Additionally, EIKD achieves an average improvement of 3.16% in student accuracy on the CIFAR-100 experiment, surpassing the classic Knowledge Distillation method by an average of 1.45%.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Eikd: method of extracting important information from feature map for knowledge distillation

  • 期刊论文

  • 发表日期：

    2025年6月23日

查看全文
  • Banghui Zhang,
  • Weiwei Zhang,
  • Liangtai Zhou,
  • Yufeng Guo,
  • Junhuang Wang,
  • Jianqing Zhu

摘要

Previous knowledge distillation research has primarily focused on transferring information from the teacher network’s final layer, often neglecting the valuable guidance embedded within the intermediate layers. This oversight can lead to a lack of direction for the student model in focusing on crucial knowledge aspects. Therefore, this paper investigates methods for effectively extracting guidance information from the teacher network’s intermediate layers and proposes the method of extracting important information from feature map for knowledge distillation (EIKD). EIKD utilizes a rank-based pruning method to extract matching guidance information from the teacher network’s intermediate layer feature map based on the size of the student’s feature map. This approach addresses the challenge of knowledge transfer caused by mismatched feature map sizes between the teacher and student models. Furthermore, to prevent the loss of correlation information between the teacher’s feature map channels during the pruning process, EIKD establishes a correlation information matrix for the extracted feature maps and guides the student to fit the relationships between the teacher’s intermediate feature map channels. This ensures that the student effectively receives both the guidance information and the feature map correlation information from the teacher’s intermediate layers. In image classification tasks, using ResNet-34 as the teacher, our method improves the ImageNet Top-1 accuracy of ResNet18 by an average of 2.14%. Additionally, EIKD achieves an average improvement of 3.16% in student accuracy on the CIFAR-100 experiment, surpassing the classic Knowledge Distillation method by an average of 1.45%.