Scholarlink

<p>Deep neural networks (DNNs), due to their high computational demands, often struggle in resource-constrained environments. Knowledge distillation is an effective model compression technique that transfers knowledge from a large teacher model to a lightweight student model, thereby improving the student’s performance. However, traditional methods face two major challenges. First, they overlook differences in sample difficulty and apply a uniform distillation strategy to all samples, which limits the student model’s ability to effectively learn from the teacher’s predictive distributions. Second, during self-distillation, shallow-layer noise and inconsistent predictions across branches can introduce misleading signals. To address these issues, this paper proposes a Dual-Path Collaborative Distillation (DPCD) framework. A dynamic weight decoupling distillation mechanism is introduced to quantify sample difficulty based on the confidence distribution of the teacher model. It adaptively adjusts the loss weights for target and non-target classes according to quantile thresholds, thus improving the student model’s learning efficiency. Additionally, a cross-branch consensus enhancement strategy filters out low-confidence noise by selecting high-consensus knowledge from dual-branch predictions. A multi-exit architecture is further employed to fuse the teacher’s global semantic information with the student’s local feature representations. Experiments on CIFAR-100 and ImageNet demonstrate that the DPCD framework outperforms mainstream knowledge distillation methods.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Dual-path collaborative distillation

  • 期刊论文

  • 发表日期：

    2026年1月29日

查看全文
  • Shihui Wang,
  • Xinwei Li,
  • Bingfeng Li,
  • Yi Yang

摘要

Deep neural networks (DNNs), due to their high computational demands, often struggle in resource-constrained environments. Knowledge distillation is an effective model compression technique that transfers knowledge from a large teacher model to a lightweight student model, thereby improving the student’s performance. However, traditional methods face two major challenges. First, they overlook differences in sample difficulty and apply a uniform distillation strategy to all samples, which limits the student model’s ability to effectively learn from the teacher’s predictive distributions. Second, during self-distillation, shallow-layer noise and inconsistent predictions across branches can introduce misleading signals. To address these issues, this paper proposes a Dual-Path Collaborative Distillation (DPCD) framework. A dynamic weight decoupling distillation mechanism is introduced to quantify sample difficulty based on the confidence distribution of the teacher model. It adaptively adjusts the loss weights for target and non-target classes according to quantile thresholds, thus improving the student model’s learning efficiency. Additionally, a cross-branch consensus enhancement strategy filters out low-confidence noise by selecting high-consensus knowledge from dual-branch predictions. A multi-exit architecture is further employed to fuse the teacher’s global semantic information with the student’s local feature representations. Experiments on CIFAR-100 and ImageNet demonstrate that the DPCD framework outperforms mainstream knowledge distillation methods.