Inter-image Discrepancy Knowledge Distillation for Semantic Segmentation

As a typical dense prediction task, semantic segmentation remains challenging in industrial automation, since it is non-trivial to achieve a good tradeoff between the performance and the efficiency. Meanwhile, knowledge distillation has been applied to reduce the computational cost in semantic segmentation task. However, existing knowledge distillation methods for semantic segmentation mainly mimic the teachers’ behaviour using the well-designed knowledge variants from a single image, failing to explore discrepancy knowledge between different images. Considering that the large pre-trained teacher network usually tends to form a more robust discrepancy space than the small student, we propose a new inter-image discrepancy knowledge distillation method (IIDKD) for semantic segmentation. Extensive experiments are conducted on two popular semantic segmentation datasets, where the experimental results show the efficiency and effectiveness of distilling inter-image discrepancy knowledge.