Scholarlink

<p>The rapid advancement of deep convolutional networks has made them a crucial method for solving various computer vision tasks. In response to the increasing demand for low-error performance in visual applications, numerous techniques have been proposed to enhance the capabilities of deep convolutional networks. As a specialized form of knowledge distillation, self-distillation (SD) eliminates the need for a separate teacher model by incorporating auxiliary branches into the intermediate layers of the network and leveraging its own supervisory signals for training. However, existing SD methods still face challenges, including insufficient utilization of shallow branch information and significant representation differences between deep and shallow branches, which limit the effectiveness of knowledge transfer. To address these issues, this paper proposes a multilevel integrated supervision self-distillation (MLISD) method, which integrates shallow classifiers’ response supervision to the deepest classifier with the feature map supervision of deeper branches toward adjacent shallower branches, thereby enriching the learned knowledge. Additionally, we introduce a parallel channel–spatial adaptive attention module and propose an improved classifier design. Comprehensive experiments conducted with multiple classical deep learning models across diverse public datasets confirm that the proposed MLISD method effectively breaks through the performance limitations of traditional self-distillation approaches. Notably, MLISD-equipped models achieve average accuracy improvements of 2.87% on CIFAR-100, 2.30% on TinyImageNet, 3.49% on CUB-200, and 4.10% on MIT-67, achieving substantial improvements over other cutting-edge self-distillation methods.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A multilevel integrated supervision self-distillation method

  • 期刊论文

  • 发表日期：

    2025年7月21日

查看全文
  • Yuxiang Wang,
  • Xuemei Lei

摘要

The rapid advancement of deep convolutional networks has made them a crucial method for solving various computer vision tasks. In response to the increasing demand for low-error performance in visual applications, numerous techniques have been proposed to enhance the capabilities of deep convolutional networks. As a specialized form of knowledge distillation, self-distillation (SD) eliminates the need for a separate teacher model by incorporating auxiliary branches into the intermediate layers of the network and leveraging its own supervisory signals for training. However, existing SD methods still face challenges, including insufficient utilization of shallow branch information and significant representation differences between deep and shallow branches, which limit the effectiveness of knowledge transfer. To address these issues, this paper proposes a multilevel integrated supervision self-distillation (MLISD) method, which integrates shallow classifiers’ response supervision to the deepest classifier with the feature map supervision of deeper branches toward adjacent shallower branches, thereby enriching the learned knowledge. Additionally, we introduce a parallel channel–spatial adaptive attention module and propose an improved classifier design. Comprehensive experiments conducted with multiple classical deep learning models across diverse public datasets confirm that the proposed MLISD method effectively breaks through the performance limitations of traditional self-distillation approaches. Notably, MLISD-equipped models achieve average accuracy improvements of 2.87% on CIFAR-100, 2.30% on TinyImageNet, 3.49% on CUB-200, and 4.10% on MIT-67, achieving substantial improvements over other cutting-edge self-distillation methods.