<p>Guided image super-resolution (GISR) aims to enhance the resolution of a low-resolution (LR) target image with the aid of another HR guidance image modality. Practical fields like medical imaging and remote sensing frequently rely on the use of different sensors to capture images of the same scene, leading to unavoidable misalignments between target and guidance image modalities. Most existing GISR approaches, including both model-driven and deep learning techniques, seldom account for these intrinsic misalignments, which leads to reduced performance when handling real-world data. To address the misalignment issue in GISR, this paper proposes a deep unrolling network based on a novel GISR objective function with alignment embedding, dubbed as DUAE-Net. Motivated by the principles of optical flow estimation, we introduce a new observation model between target and guidance images and then simultaneously formulate the alignment and super-resolution in a unified framework. The optimization problem is minimized by unrolling the iterative optimization process, which is reformulated as a deep convolutional network. A dual-domain Unet (Unet-DD) module which exploits local and global information in parallel is designed to approximate the proximal operation associated with the implicit HR target image prior. Beyond using the single HR image as supervision, a consistency loss is formulated to regulate both spatial degradation and spectral transformation during training. By introducing consistency loss, the model gains extra constraints that improve HR image estimation and further strengthen the effectiveness and realism of both the spatial degradation and spectral transformation modules. Experimental results at three classical GISR tasks are provided to highlight the effectiveness of the proposed approach compared with other state-of-the-art (SOTA) techniques.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Deep Algorithm Unrolling with Alignment Embedding for Guided Image Super-resolution

  • Faming Fang,
  • Tingting Wang,
  • Junkang Zhang,
  • Zhongyang Li,
  • Aimin Zhou,
  • Riquan Zhang,
  • Guixu Zhang

摘要

Guided image super-resolution (GISR) aims to enhance the resolution of a low-resolution (LR) target image with the aid of another HR guidance image modality. Practical fields like medical imaging and remote sensing frequently rely on the use of different sensors to capture images of the same scene, leading to unavoidable misalignments between target and guidance image modalities. Most existing GISR approaches, including both model-driven and deep learning techniques, seldom account for these intrinsic misalignments, which leads to reduced performance when handling real-world data. To address the misalignment issue in GISR, this paper proposes a deep unrolling network based on a novel GISR objective function with alignment embedding, dubbed as DUAE-Net. Motivated by the principles of optical flow estimation, we introduce a new observation model between target and guidance images and then simultaneously formulate the alignment and super-resolution in a unified framework. The optimization problem is minimized by unrolling the iterative optimization process, which is reformulated as a deep convolutional network. A dual-domain Unet (Unet-DD) module which exploits local and global information in parallel is designed to approximate the proximal operation associated with the implicit HR target image prior. Beyond using the single HR image as supervision, a consistency loss is formulated to regulate both spatial degradation and spectral transformation during training. By introducing consistency loss, the model gains extra constraints that improve HR image estimation and further strengthen the effectiveness and realism of both the spatial degradation and spectral transformation modules. Experimental results at three classical GISR tasks are provided to highlight the effectiveness of the proposed approach compared with other state-of-the-art (SOTA) techniques.