Road object detection is particularly challenging in extreme weather conditions such as rain, fog, cloudy and snow, as these conditions degrade image quality and consequently undermine recognition accuracy. To address this issue, we propose YOLO-PFD, a visible and polarization images fusion target detection algorithm based on YOLOv11. The algorithm mainly improves from three aspects. First, a Dual-Modal Fusion Module (DFM) is designed in the front-end fusion stage. This module can dynamically integrate the complementary information from visible and polarization images and maximally preserves the detailed features of different modalities. Second, in the feature extraction part, the Bottleneck in C3K2 is replaced with the Gaussian Enhancement Module (GEM). This module uses Gaussian function to suppress noise in degraded images and enhance object edges, thereby improving the network’s robustness to low-quality feature maps. Third, in the detection head, a Localization Quality Evaluation (LQE) combined with shared convolution is introduced. This approach not only reduces the number of parameters but also reduces the inconsistency between classification and localization. Experiments on the polarization road object detection datasets show that the mAP@0.5 of the proposed method reaches 94.0%. Moreover, it performs excellently in terms of generalization and robustness, verifying its practical value in extreme weather scenarios.