This paper investigates the fusion based on luminance map of visible degree of polarization and infrared intensity map. Based on the Stokes parameters, the inputs are the \(S_{0}\) image for infrared light and the degree of polarization image for visible light. Meanwhile, the rolling guidance filter is used to decompose source images to obtain the detail and base layers. For the base layer fusion, to enhance the visual contrast, an approximate luminance map is proposed as the fusion weights of the base layer. To fuse detail images, we propose an improved weighted least squares to retain more details. We created a polarized image dataset. The results show that our method gets higher scores in quantitative calculation, which shows that our fused images have higher quality compared to other methods.