Multispectral object detection using both RGB and infrared (IR) data improves accuracy and robustness by fusing complementary information from different modalities. However, current methods lack adaptability to real-world application scenarios and are not conducive to practical deployment. Therefore, we propose a fast multimodal detection based on prior scene repair for all-weather road sensing (PSRDET): Firstly, we design a Cross-Visual State Space Fusion (CVSSF) module that maps cross-modal features to the hidden state space for asymptotic interaction. Facing the challenges of various complex environments, we propose the prior-based degradation-resistant hybrid expert (PDR-MoE) module to adaptively adjust the visible images to mitigate the image degradation problem caused by harsh environments. Finally, the module is embedded into our dual-stream backbone network. Experiments show that PSRDET achieves state-of-the-art performance on FLIR and M3FD benchmark datasets, thus offering greater safety and environmental adaptability in the field of autonomous driving.