We propose a score-aware evolutionary knowledge distillation (SAEKD) framework for training compact student networks using a multi-teacher setup guided by score-aware loss modulation, peer collaboration, and adaptive ensemble supervision. Unlike traditional KD pipelines that rely on static teacher-student pairings, our method dynamically selects teachers based on a performance ranking, modulates the distillation temperature accordingly, and blends signals from peer and top-performing teachers. The students evolve over epochs through curriculum-based training and survival selection, promoting stability and diversity. Experiments on CIFAR-10 and CIFAR-100 demonstrate that our lightweight student models, trained with as few as 2.7M and 3.1M parameters, achieve up to 80.75% and 54.11% test accuracy, respectively, outperforming conventional KD baselines and competitive student architectures at similar model sizes. SAEKD offers a robust and adaptive approach to scalable knowledge distillation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Score-Aware Evolutionary Knowledge Distillation via Multi-teacher Supervision and Peer Collaboration

  • Divyansh Bhatia

摘要

We propose a score-aware evolutionary knowledge distillation (SAEKD) framework for training compact student networks using a multi-teacher setup guided by score-aware loss modulation, peer collaboration, and adaptive ensemble supervision. Unlike traditional KD pipelines that rely on static teacher-student pairings, our method dynamically selects teachers based on a performance ranking, modulates the distillation temperature accordingly, and blends signals from peer and top-performing teachers. The students evolve over epochs through curriculum-based training and survival selection, promoting stability and diversity. Experiments on CIFAR-10 and CIFAR-100 demonstrate that our lightweight student models, trained with as few as 2.7M and 3.1M parameters, achieve up to 80.75% and 54.11% test accuracy, respectively, outperforming conventional KD baselines and competitive student architectures at similar model sizes. SAEKD offers a robust and adaptive approach to scalable knowledge distillation.