Scholarlink

<p>Data-Free Knowledge Distillation (DFKD) is a good approach for model compression in privacy-constrained settings. Recent paradigms moving to Denoising Diffusion Probabilistic Models (DDPM) from GANs over time, have seen high accuracy for the synthetic data representation but are computationally expensive due to thousands of rounds of iterative sampling steps. In Fast-AD, we introduce a novel framework of Fast-AD for Denoising Diffusion Implicit Models (DDIM) that allow it to accelerate synthesis by several orders of magnitude over state-of-the-art approaches. However, it is clear that a critical issue in terms of step size would be to integrate conventional fixed-strength gradient instructions (for example, BN statistics and Energy-based priors) on the aggressive large-step sampling of DDIM, where trajectory overshooting and severe mode collapse is not infrequent. This is an enigma that we solve with Confidence-Aware Dynamic Rectification (CADR). Unlike static methods, CADR functions as a closed-loop controller that adapts intensity of guidance dynamically based on the level of uncertainty in the teacher model during real-time prediction. It also has an adaptive gradient normalization to prevent structural damage early on during denoising. This "intervention-on-demand" mechanism has been able to adjust the synthetic distribution to the training manifold of the teacher, while the variety of diffusion models remains in the original set. A large series of experiments on CIFAR-100 benchmarks and ImageNet prove that Fast-AD reaches top distillation accuracy with 20<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\times \)</EquationSource> </InlineEquation> less synthesis time, which means an efficient effect as well. Code is available at: <a href="https://github.com/jcynxu/fast-AD">https://github.com/jcynxu/fast-AD</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Fast and faithful: accelerating data-free knowledge distillation via confidence-aware adaptive diffusion

  • 期刊论文

  • 发表日期：

    2026年2月17日

查看全文
  • Chenyang Jiang,
  • Zhendong Li,
  • Shuai Li,
  • Hao Liu

摘要

Data-Free Knowledge Distillation (DFKD) is a good approach for model compression in privacy-constrained settings. Recent paradigms moving to Denoising Diffusion Probabilistic Models (DDPM) from GANs over time, have seen high accuracy for the synthetic data representation but are computationally expensive due to thousands of rounds of iterative sampling steps. In Fast-AD, we introduce a novel framework of Fast-AD for Denoising Diffusion Implicit Models (DDIM) that allow it to accelerate synthesis by several orders of magnitude over state-of-the-art approaches. However, it is clear that a critical issue in terms of step size would be to integrate conventional fixed-strength gradient instructions (for example, BN statistics and Energy-based priors) on the aggressive large-step sampling of DDIM, where trajectory overshooting and severe mode collapse is not infrequent. This is an enigma that we solve with Confidence-Aware Dynamic Rectification (CADR). Unlike static methods, CADR functions as a closed-loop controller that adapts intensity of guidance dynamically based on the level of uncertainty in the teacher model during real-time prediction. It also has an adaptive gradient normalization to prevent structural damage early on during denoising. This "intervention-on-demand" mechanism has been able to adjust the synthetic distribution to the training manifold of the teacher, while the variety of diffusion models remains in the original set. A large series of experiments on CIFAR-100 benchmarks and ImageNet prove that Fast-AD reaches top distillation accuracy with 20 \(\times \) less synthesis time, which means an efficient effect as well. Code is available at: https://github.com/jcynxu/fast-AD.