Scholarlink

<p>Model compression is critical for deploying deep learning models on resource-constrained devices. We introduce a novel method enhancing knowledge distillation with Integrated Gradients (IG) as a data augmentation strategy. Our approach overlays IG maps onto input images during training, providing student models with deeper insights into teacher models’ decision-making processes. Extensive evaluation on CIFAR-10 demonstrates that our IG-augmented knowledge distillation achieves 92.6% testing accuracy with a 4.1x compression factor–a significant 1.1 percentage point improvement (p &lt; 0.001) over non-distilled models (91.5%). This compression reduces inference time from 140ms to 13ms. Our method precomputes IG maps before training, transforming substantial runtime costs into a one-time preprocessing step. We validate our approach through systematic ablation studies with attention transfer, comprehensive compression factor analysis (2.2<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\times\)</EquationSource> </InlineEquation>–1122<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\times\)</EquationSource> </InlineEquation>), Monte Carlo simulations, and cross-dataset evaluation on ImageNet subsets, demonstrating that IG-based knowledge distillation consistently outperforms conventional approaches. Our results establish this framework as a viable compression technique for real-world deployment on edge devices while maintaining competitive accuracy.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Model compression using knowledge distillation with integrated gradients

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月28日

查看全文
  • David E. Hernandez,
  • Jose Ramon Chang,
  • Torbjörn E. M. Nordling

摘要

Model compression is critical for deploying deep learning models on resource-constrained devices. We introduce a novel method enhancing knowledge distillation with Integrated Gradients (IG) as a data augmentation strategy. Our approach overlays IG maps onto input images during training, providing student models with deeper insights into teacher models’ decision-making processes. Extensive evaluation on CIFAR-10 demonstrates that our IG-augmented knowledge distillation achieves 92.6% testing accuracy with a 4.1x compression factor–a significant 1.1 percentage point improvement (p < 0.001) over non-distilled models (91.5%). This compression reduces inference time from 140ms to 13ms. Our method precomputes IG maps before training, transforming substantial runtime costs into a one-time preprocessing step. We validate our approach through systematic ablation studies with attention transfer, comprehensive compression factor analysis (2.2 \(\times\) –1122 \(\times\) ), Monte Carlo simulations, and cross-dataset evaluation on ImageNet subsets, demonstrating that IG-based knowledge distillation consistently outperforms conventional approaches. Our results establish this framework as a viable compression technique for real-world deployment on edge devices while maintaining competitive accuracy.