A dual-modal semantic guidance and differential feature complementation fusion method for infrared and visible image

The purpose of infrared and visible image fusion is to synthesize a single image with rich details using the complementary information of the two modal images. Unlike current research methods which primarily focus on the visual quality of the fused images, our method emphasizes the importance of image fusion in enhancing downstream tasks. We propose a dual-modal semantic guidance strategy, which uses a dual-branch semantic segmentation network to guide the fusion network. Specifically, our method utilizes the segmentation results of the infrared and visible images to calculate the mean intersection over union and adjusts the loss function accordingly to guide the fusion network. Additionally, we introduce a novel component called the differential feature complementation module, which strengthen the fusion of complementary information by computing and integrating differential features at the same level of the fusion network. Comparison experiments on the MFNet dataset demonstrate that our method outperforms the performance of existing state-of-the-art methods in terms of fused image quality. Furthermore, segmentation experiments on the MFNet dataset demonstrate that our method effectively improves the performance in the context of the semantic segmentation task. Generalization experiments on the TNO and RoadScene datasets demonstrate that our method also possesses the strong generalization capability.