Repdistiller: Knowledge Distillation Scaled by Re-parameterization for Crowd Counting

Knowledge distillation (KD) is an important method to compress a large teacher model into a much smaller student model. However, the large capacity gap between the teacher and student models hinders the performance of KD in various tasks. In this paper, we propose Repdistiller, a knowledge distillation framework combined with structural re-parameterization to alleviate the capacity gap problem. Repdistiller makes the student model search for parallel branches during training, thus the capacity gap between the teacher and student models is decreased. After knowledge distillation, the searched branches are merged into the student network without causing any computation overhead for inference. Taking the crowd counting task as an example, Repdistiller achieves state-of-the-art performance on the ShanghaiTech and UCF-QNRF datasets, outperforming many well-established knowledge distillation methods.