One of the challenges in image fusion is the unbalanced image qualities of source images, which seriously affects the fusion effect. Here we present an adaptive fusion technique for the combination of infrared and visible images, called ADFuse. Firstly, we propose an adaptive mechanism, which can automatically adjust the optimization objectives to train in accordance with the qualities of the source images, thereby enhancing the feature extraction ability and robustness of the network. Then we regard image fusion as the unity of detail maintenance problem and intensity maintenance problem, we segment the loss function into three distinct components to surprise the network preserve feature information effectively. Simultaneously, we combine the adaptive mechanism with the loss function to supervise the optimization process of ADFuse. A large number of experimental data from two publicly available datasets confirm the competitiveness and performance of our method.