Image Inpainting with Semantic U-Transformer

With the driving force of powerful convolutional neural networks, image inpainting has made tremendous progress. Recently, transformer has demonstrated its effectiveness in various vision tasks, mainly due to its capacity to model long-term relationships. However, when it comes to image inpainting tasks, the transformer tends to fall short in terms of modeling local information, and interference from damaged regions can pose challenges. To tackle these issues, we introduce a novel Semantic U-shaped Transformer (SUT) in this work. The SUT is designed with spectral transformer blocks in its shallow layers, effectively capturing local information. Conversely, deeper layers utilize BRA transformer blocks to model global information. A key feature of the SUT is its attention mechanism, which employs bi-level routing attention. This approach significantly reduces the interference of damaged regions on overall information, making the SUT more suitable for image inpainting tasks. Experiments on several datasets indicate that the performance of the proposed method outperforms the current state-of-the-art (SOTA) inpainting approaches. In general, the PSNR of our method is on average 0.93 dB higher than SOTA, and the SSIM is higher by 0.026.