Infrared and Visible Image Fusion via Test-Time Training

Infrared and visible image fusion (IVIF) is a widely used technique in instrument-related fields. It aims at extracting contrast information from the infrared image and texture details from the visible image and combining these two kinds of information into a single image. Most auto-encoder-based methods train the network on natural images, such as MS-COCO, and test the model on IVIF datasets. This kind of method suffers from domain shift issues and cannot generalize well in real-world scenarios. To this end, we propose a self-supervised test-time training (TTT) approach to facilitate learning a better fusion result. Specifically, a new self-supervised loss is developed to evaluate the quality of the fusion result. This loss function directs the network to improve the fusion quality by optimizing model parameters with a small number of iterations in the test time. Besides, instead of manually designing fusion strategies, we leverage a fusion adapter to automatically learn fusion rules. Experimental comparisons on two public IVIF datasets validate that the proposed method outperforms existing methods subjectively and objectively.