DS-Diff: a dual-stage network with degradation-aware and semantic-aware for adverse weather removal based on diffusion models

Removing weather degradation is a common task in the field of image restoration. Given the success of diffusion models in low-level visual tasks, we apply them to adverse weather image restoration. Nevertheless, the current diffusion model-based restoration networks conflate degradation information with inherent image content, resulting in poor performance. To address this, we propose DS-Diff to decompose the task into two stages: degradation removal and content generation. Specifically, we first employ the degradation-aware module to eliminate weather disturbance prospects by transforming the representation to the frequency domain via the Fourier theory for low computation. Then, we freeze the parameters of the module, and propose the Semantic-aware generation module to generate image content lost in degradation removal process, which exploits the generation ability of diffusion models while using semantic features as condition. Furthermore, we introduce global structural refinement into generation module to mitigate structural inconsistencies that may occur in the reverse process, significantly improving the performance. We empirically evaluate our model on multiple benchmark datasets of various weather degradation. Extensive experiments validate the rationality and superiority of our DS-Diff approach.