Image restoration in harsh weather conditions is a key challenge in computer vision. Most of the existing methods mainly focus on modeling pixels and ignore the frequency aspect. Additionally, most of the existing restoration networks use multi-scale structures, but the restoration defects they introduce to the detail information limit their performance in recovering image details. In this work, we propose WIRMamba, a frequency-domain fusion model based on a selective structured state space model. In concrete terms, we innovatively design the Frequency State Space Module and a Multi-scale Wavelet Fusion Module. They achieve a synergistic analysis of the dynamic and frequency characteristics of the input signal by complementarily integrating the state space model and the Fourier Transform. The simultaneous combination with a learnable Multi-scale Wavelet Fusion Module emphasizes the directional continuity and frequency features during the gradual transition from degraded to clear images. Compared with weather-specific methods, our method achieves significant performance metrics on multiple datasets, highlighting the importance of long sequence modeling and frequency information in image restoration.