Scholarlink

<p>Most existing self-knowledge distillation methods achieve self-learning by transferring knowledge from independent auxiliary branches. However, their learning ability is often limited due to the isolated branches. Furthermore, the limited local features are insufficient for shallow layers to learn richer features during knowledge transfer. To address these issues, this paper proposes a novel self-knowledge distillation framework called Elite Teacher Self-Knowledge Distillation (ETSKD). Specifically, multiple auxiliary branches are attached to different stages of the backbone network. While calculating the soft probability distillation loss at the output layer, the logits of the auxiliary branches are guided by an elite teacher obtained through dynamic weighting based on entropy uncertainty. For intermediate feature distillation, the spatial-frequency fusion module enhances feature representations, where the deepest auxiliary branch serves as the teacher to propagate refined spatial-frequency features to the shallower ones. Extensive experiments on CIFAR-100 and ImageNet-2012 demonstrate that ETSKD improves classification accuracy and outperforms most state-of-the-art self-distillation approaches. Our codes are available at: <a href="https://github.com/djiangkai/ETSKD">https://github.com/djiangkai/ETSKD</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ETSKD: self-knowledge distillation via Elite teacher and Spatial-frequency fusion feature

  • 期刊论文

  • 发表日期：

    2025年10月25日

查看全文
  • Jiangkai Du,
  • Xinwei Li

摘要

Most existing self-knowledge distillation methods achieve self-learning by transferring knowledge from independent auxiliary branches. However, their learning ability is often limited due to the isolated branches. Furthermore, the limited local features are insufficient for shallow layers to learn richer features during knowledge transfer. To address these issues, this paper proposes a novel self-knowledge distillation framework called Elite Teacher Self-Knowledge Distillation (ETSKD). Specifically, multiple auxiliary branches are attached to different stages of the backbone network. While calculating the soft probability distillation loss at the output layer, the logits of the auxiliary branches are guided by an elite teacher obtained through dynamic weighting based on entropy uncertainty. For intermediate feature distillation, the spatial-frequency fusion module enhances feature representations, where the deepest auxiliary branch serves as the teacher to propagate refined spatial-frequency features to the shallower ones. Extensive experiments on CIFAR-100 and ImageNet-2012 demonstrate that ETSKD improves classification accuracy and outperforms most state-of-the-art self-distillation approaches. Our codes are available at: https://github.com/djiangkai/ETSKD.