SSRT: Intra- and cross-view attention for stereo image super-resolution

Stereo image super-resolution (SR) aims to increase the spatial resolutions of the left and right views of a stereo image in an attempt to generate higher-resolutions views that appear visually equivalent to those obtained with a higher-resolution stereo camera. The field of stereo image SR has seen rapid significant progress due in large part to the application of deep-learning-based techniques and its associated recent research advancements. Yet, despite this progress, stereo images captured under real-world conditions (e.g., using consumer-level cameras in non-laboratory settings) often contain irregular disparities between the left and right views, a fact that has not been fully considered nor properly addressed in previous works. To address this issue, in this paper, we propose a stereo image super-resolution Transformer (SSRT) network which consists of two blocks, a multi-kernel Transformer block and a cross-merging block, to fully extract intra-view features and capture cross-view dependencies. The multi-kernel Transformer block is proposed to increase the number of representation subspaces for intra-view feature extraction. The cross-merging block utilizes patch-wise attention which efficiently expands the search area to tackle stereo image pairs with arbitrary pixel offsets. Experimental results demonstrate that, for 2 \(\times \) $\times$ stereo image super-resolution, our model with a comparable number of network parameters achieves 37.57 dB on ETH3D, 35.88 dB on Middlebury, 29.56 dB on Flickr1024, 31.52 dB on KITTI 2012, and 31.15 dB on KITTI2015 in terms of PSNR, and surpasses the state-of-the-art method by a large margin of +0.80 dB on ETH3D, +0.57 dB on Middlebury, +0.36 dB on Flickr1024, +0.14 dB on KITTI 2012, and +0.06 dB on KITTI 2015. The code is available at https://github.com/yanksx233/SSRT.

Network architecture of the proposed SSRT model, which consists of multi-kernel Transformer blocks (MKTBs) and crossmerging blocks (CMBs) alternately concatenated. The MKTB employs the neighborhood attention (positional adaptive convolution) and the transposed attention (regional adaptive convolution) for more effective intra-view feature extraction. The CMB employs the patch-wise attention for more effective modeling of the cross-view dependencies between the two views with irregular disparities