Image restoration is a crucial low-level task in computer vision, aimed at recovering degraded images, such as those affected by rain and fog. Recently, Transformer-based models have achieved remarkable success in image restoration tasks. However, these models inevitably introduce redundant information, which degrades the quality of outputs. Additionally, Vision Transformers are often insensitive to high-frequency details. To address these, we propose a novel image restoration model named Frequency-Sensitive Sparse Transformer (FSST), which includes Frequency-Sensitive Sparse Attention (FSSA) and Multi-Granularity Refinement Network (MGRN). Specifically, FSSA uses a ReLU-based sparse self-attention mechanism to maintain the most informative features, while a dense self-attention compensates for the information loss caused by sparse self-attention. Additionally, we design a Fast Fourier Transform Resblock in FSSA to enhance the model’s sensitivity to frequency domain information. On the other hand, MGRN integrates information at multiple granularities within the feature maps, which further enhances FSSA’s sensitivity to high-frequency details. Extensive experimental results demonstrate the effectiveness of our method in restoring a clear image. The code is available at https://github.com/LaughingHH/FSST-main .