Feature-level knowledge distillation is a promising approach for compressing object detectors. It transfers intermediate representations from high-capacity, computationally intensive teacher models to lightweight students with lower inference costs. However, existing methods predominantly emphasize spatial and channel-wise alignment, while largely overlooking the frequency characteristics that can be derived from deep features via spectral decomposition. In this work, we present a frequency-aware knowledge distillation approach that enhances existing frameworks through the integration of wavelet-based feature alignment. Specifically, teacher features are decomposed into multi-level subbands using two-dimensional Haar wavelet transforms, enabling the student to perform subband-wise alignment that captures both high-frequency details and low-frequency semantic cues. Additionally, a relational distillation mechanism operating on the principal subband is employed to model global dependencies and enhance semantic consistency. To evaluate the effectiveness of the proposed approach, we conduct experiments on the Inspection of Power Line Assets Dataset (InsPLAD), a real-world UAV dataset for transmission infrastructure inspection, using various detectors including both two-stage and one-stage models. Building upon the FGD framework, our method yields distilled lightweight student models that consistently achieve 1%–3% gains in average precision (AP) across various detectors. Furthermore, it demonstrates superior detection performance, particularly in handling elongated and structurally complex objects that typically pose challenges for compact models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Wavelet-Based Distillation with Structured Frequency Alignment

  • Pengyu Lu,
  • Junfei Yi,
  • Jianxu Mao,
  • Junlong Yu,
  • Shuohao Xiao,
  • Zhenyu He,
  • Yaonan Wang

摘要

Feature-level knowledge distillation is a promising approach for compressing object detectors. It transfers intermediate representations from high-capacity, computationally intensive teacher models to lightweight students with lower inference costs. However, existing methods predominantly emphasize spatial and channel-wise alignment, while largely overlooking the frequency characteristics that can be derived from deep features via spectral decomposition. In this work, we present a frequency-aware knowledge distillation approach that enhances existing frameworks through the integration of wavelet-based feature alignment. Specifically, teacher features are decomposed into multi-level subbands using two-dimensional Haar wavelet transforms, enabling the student to perform subband-wise alignment that captures both high-frequency details and low-frequency semantic cues. Additionally, a relational distillation mechanism operating on the principal subband is employed to model global dependencies and enhance semantic consistency. To evaluate the effectiveness of the proposed approach, we conduct experiments on the Inspection of Power Line Assets Dataset (InsPLAD), a real-world UAV dataset for transmission infrastructure inspection, using various detectors including both two-stage and one-stage models. Building upon the FGD framework, our method yields distilled lightweight student models that consistently achieve 1%–3% gains in average precision (AP) across various detectors. Furthermore, it demonstrates superior detection performance, particularly in handling elongated and structurally complex objects that typically pose challenges for compact models.