Scholarlink

<p>Existing knowledge distillation (KD) methods focus on constructing efficient spatial knowledge representations for student networks, which often neglect the temporal nature of knowledge transfer in KD. In this paper, we show empirically that using static loss function weights limits the students’ ability to learn over time, and that focusing on low-noise tasks early in training facilitates their learning process. Building on these observations, we propose a temporal curriculum-based knowledge distillation method, which we call Dynamic Curriculum Knowledge Distillation (DCKD). DCKD comprises two complementary components: Adaptive Distillation Weight (ADW) and Crescendo Adversarial Distillation (CAD). In early training stages, ADW introduces a learnable task noise parameter based on homoscedastic uncertainty to measure task difficulty and automatically adjust weights, guiding the student network to focus on low-noise tasks. As training progresses, CAD leverages this same learnable noise parameter to reverse gradients, shifting the curriculum from simpler to more challenging tasks through an adversarial mechanism. The learnable noise parameter thus serves as a bridge connecting ADW and CAD, enabling smooth curriculum progression. As a general integration framework, DCKD can be seamlessly integrated into state-of-the-art (SOTA) knowledge distillation frameworks. Comprehensive evaluations on CIFAR-100, ImageNet, and MS-COCO benchmarks indicate that our approach significantly improves performance across ten SOTA knowledge distillation methods, yielding competitive results. Our code is available at <a href="https://github.com/ITVR-lab/DCKD.git">https://github.com/ITVR-lab/DCKD.git</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Dynamic curriculum knowledge distillation: optimizing knowledge transfer through temporal adaptation

  • 期刊论文

  • 发表日期：

    2026年2月3日

查看全文
  • Huaping Zhou,
  • Jin Wu,
  • Xiangrui Meng,
  • Kelei Sun,
  • Tao Wu,
  • Bing Deng

摘要

Existing knowledge distillation (KD) methods focus on constructing efficient spatial knowledge representations for student networks, which often neglect the temporal nature of knowledge transfer in KD. In this paper, we show empirically that using static loss function weights limits the students’ ability to learn over time, and that focusing on low-noise tasks early in training facilitates their learning process. Building on these observations, we propose a temporal curriculum-based knowledge distillation method, which we call Dynamic Curriculum Knowledge Distillation (DCKD). DCKD comprises two complementary components: Adaptive Distillation Weight (ADW) and Crescendo Adversarial Distillation (CAD). In early training stages, ADW introduces a learnable task noise parameter based on homoscedastic uncertainty to measure task difficulty and automatically adjust weights, guiding the student network to focus on low-noise tasks. As training progresses, CAD leverages this same learnable noise parameter to reverse gradients, shifting the curriculum from simpler to more challenging tasks through an adversarial mechanism. The learnable noise parameter thus serves as a bridge connecting ADW and CAD, enabling smooth curriculum progression. As a general integration framework, DCKD can be seamlessly integrated into state-of-the-art (SOTA) knowledge distillation frameworks. Comprehensive evaluations on CIFAR-100, ImageNet, and MS-COCO benchmarks indicate that our approach significantly improves performance across ten SOTA knowledge distillation methods, yielding competitive results. Our code is available at https://github.com/ITVR-lab/DCKD.git.