Cross-Weighting Knowledge Distillation for Object Detection

Knowledge distillation(KD) has been widely utilized for compressing object detection models and enhancing their accuracy. However, most current knowledge distillation methods have not adequately addressed common issues in object detection, including the severe imbalance between foreground and background samples, and the discrepancy in integrating classification scores and IoU between the detector’s training and testing phases. In this paper, we propose the Cross-Weighting Knowledge Distillation(CWKD) method, where localization and classification are jointly considered during the distillation training phase. Within this framework, we introduce two specific losses, namely, the IoU-aware Classification KD Loss and the Class-aware Localization KD Loss. We also propose the Valuable Distillation Object(VDO) selection module, aimed at discovering additional objects beyond the ground truth objects that are worth distillation, thereby further expanding positive samples to address the imbalance issue. Utilizing the logit-based knowledge distillation approach, our method can seamlessly integrate into both single-stage and multi-stage object detection models. Through extensive experiments on the challenging object detection dataset COCO, we demonstrate our effectiveness and surpass other logit-based and feature-based knowledge distillation methods.