Image restoration aims to obtain a high-quality image from a degraded one. For real-world applications, an increasing number of methods are moving towards addressing multiple degradations using a single model. However, most of these methods still require task-specific training and primarily extract information from the spatial domain. To overcome this challenge, we introduce a novel All-in-one network, FASPNet, which effectively incorporates both frequency and spatial information to handle various degradations, without requiring any degradation priors. Specifically, we propose a Frequency Refiner Module (FRM), which adaptively adjusts frequency representations and captures crucial global frequency information to facilitate better image restoration. Furthermore, to provide essential low-level information related to restoration, we introduce a Spatial Prompt Module (SPM), utilizing prompts to encode restoration-relevant spatial detail representations and abstract degradation patterns. Extensive experiments have demonstrated that our model outperforms other baseline models on multiple datasets for three common and challenging tasks: deraining, dehazing, and denoising.