Wavelet-Based Distillation with Structured Frequency Alignment
摘要
Feature-level knowledge distillation is a promising approach for compressing object detectors. It transfers intermediate representations from high-capacity, computationally intensive teacher models to lightweight students with lower inference costs. However, existing methods predominantly emphasize spatial and channel-wise alignment, while largely overlooking the frequency characteristics that can be derived from deep features via spectral decomposition. In this work, we present a frequency-aware knowledge distillation approach that enhances existing frameworks through the integration of wavelet-based feature alignment. Specifically, teacher features are decomposed into multi-level subbands using two-dimensional Haar wavelet transforms, enabling the student to perform subband-wise alignment that captures both high-frequency details and low-frequency semantic cues. Additionally, a relational distillation mechanism operating on the principal subband is employed to model global dependencies and enhance semantic consistency. To evaluate the effectiveness of the proposed approach, we conduct experiments on the Inspection of Power Line Assets Dataset (InsPLAD), a real-world UAV dataset for transmission infrastructure inspection, using various detectors including both two-stage and one-stage models. Building upon the FGD framework, our method yields distilled lightweight student models that consistently achieve 1%–3% gains in average precision (AP) across various detectors. Furthermore, it demonstrates superior detection performance, particularly in handling elongated and structurally complex objects that typically pose challenges for compact models.