Existing networks for fusing infrared and visible images, whe-ther CNN-based or Transformer-based, struggle to effectively extract and utilize global and local features, leading to poor fusion performance and accuracy. To address this, we propose a two-branch network specifically designed to comprehensively extract and integrate the global and local features of both image types. This network comprises three main components: a global feature extraction module, a detail feature extraction module, and a feature fusion module. The global feature extraction module primarily focuses on capturing global dependencies and high-level semantic information through a self-attention mechanism. This allows it to capture long-range dependencies between different spatial locations in the image, thereby enhancing the understanding of global semantic information. Concurrently, multi-head cross-attention facilitates global interactions between the infrared and visible images, capturing their interdependencies. The detail feature extraction module targets local detail and underlying textures, integrating an Invertible Neural Networks (INN) module with the Convolutional Block Attention Module (CBAM) to ensure lossless information transmission and feature enhancement. Finally, the feature fusion module merges the global and local features based on the activity levels of the row and column vectors, effectively preserving information and minimizing feature conflicts and losses during the fusion process. Experiments on datasets demonstrate that our network outperforms most current methods in terms of metrics and visualization results, and can better preserve critical information and detail such as texture, and contour.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Two-Branch Fusion Network for Infrared and Visible Image Fusion

  • Weihao Zhang,
  • Zhilin Li,
  • Bin Li,
  • Mingliang Zhang

摘要

Existing networks for fusing infrared and visible images, whe-ther CNN-based or Transformer-based, struggle to effectively extract and utilize global and local features, leading to poor fusion performance and accuracy. To address this, we propose a two-branch network specifically designed to comprehensively extract and integrate the global and local features of both image types. This network comprises three main components: a global feature extraction module, a detail feature extraction module, and a feature fusion module. The global feature extraction module primarily focuses on capturing global dependencies and high-level semantic information through a self-attention mechanism. This allows it to capture long-range dependencies between different spatial locations in the image, thereby enhancing the understanding of global semantic information. Concurrently, multi-head cross-attention facilitates global interactions between the infrared and visible images, capturing their interdependencies. The detail feature extraction module targets local detail and underlying textures, integrating an Invertible Neural Networks (INN) module with the Convolutional Block Attention Module (CBAM) to ensure lossless information transmission and feature enhancement. Finally, the feature fusion module merges the global and local features based on the activity levels of the row and column vectors, effectively preserving information and minimizing feature conflicts and losses during the fusion process. Experiments on datasets demonstrate that our network outperforms most current methods in terms of metrics and visualization results, and can better preserve critical information and detail such as texture, and contour.