Enhancing Image Super-Resolution with Dual Compression Transformer

Transformer-based methods have demonstrated substantial advancements in image super-resolution (SR). However, their success requires a high computational cost. One reason is that the Transformer uses computationally expensive self-attention in its cascaded layers, which has quadratic computational complexity with respect to the number of input tokens. In this work, we propose a novel Transformer model, the dual compression Transformer (DCT), for image SR. Our DCT compresses self-attention computations within and between layers. Specifically, we propose the linear self-attention (L-SA) using a carefully designed linear approximation normalization function, which compresses the quadratic complexity within the layer to linear complexity by approximating the \({{\,\textrm{Softmax}\,}}\) $\text{Softmax}$ operation. Additionally, we propose the linear-window self-attention (LW-SA), which combines L-SA with window-based self-attention (W-SA) to better leverage the global context. Furthermore, we design an approximate attention module (AAM) to replace self-attention computations within the layer. To achieve compressed attention computations between layers, we alternate using LW-SA and AAM within the traditional cascade framework. Extensive experiments demonstrate that our DCT outperforms current state-of-the-art methods with superior performance and reduced computational complexity. The code is available at https://github.com/zhengchen1999/DCT.