<p>The Visual Question Answering (VQA) model generally adopts the Transformer architecture. In order to make better predictions, the model must have a strong correlation between the understanding of text and images. If image regions unrelated to the input question are iterated from low-level to high-level in the decoder, a significant amount of image noise will be generated, distracting the model’s attention and negatively impacting performance. To address this issue, this paper introduces a new architecture called Capsule Spatial Attention Mask (CSAM) network. This architecture first generates spatial positional information of grid features through the Spatial Attention Channel Enhancement Module to strengthen the sub-feature representation within the image. Then, through the Dynamic Capsule Attention Mask module outputs prompt vectors, which modify the cross attention map, by employing a gradually decreasing weight approach, CSAM retains valid image vectors in stages. Due to the presence of these two modules, CSAM can focus on high-level semantic features correctly and avoid image noise. Experimental results on the VQA 2.0 and CLEVR datasets demonstrate that the CSAM model outperforms existing state-of-the-art models. Moreover, ablation studies further analyze the impact of parameter settings on model performance, while visual examples reveal the interpretability of the model. These results validate the effectiveness and practicality of CSAM in handling visual and language multimodal tasks.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CSAM: Capsule spatial attention mask network for visual question answering

  • Lixia Xue,
  • Yi Huang,
  • Ronggui Wang,
  • Juan Yang

摘要

The Visual Question Answering (VQA) model generally adopts the Transformer architecture. In order to make better predictions, the model must have a strong correlation between the understanding of text and images. If image regions unrelated to the input question are iterated from low-level to high-level in the decoder, a significant amount of image noise will be generated, distracting the model’s attention and negatively impacting performance. To address this issue, this paper introduces a new architecture called Capsule Spatial Attention Mask (CSAM) network. This architecture first generates spatial positional information of grid features through the Spatial Attention Channel Enhancement Module to strengthen the sub-feature representation within the image. Then, through the Dynamic Capsule Attention Mask module outputs prompt vectors, which modify the cross attention map, by employing a gradually decreasing weight approach, CSAM retains valid image vectors in stages. Due to the presence of these two modules, CSAM can focus on high-level semantic features correctly and avoid image noise. Experimental results on the VQA 2.0 and CLEVR datasets demonstrate that the CSAM model outperforms existing state-of-the-art models. Moreover, ablation studies further analyze the impact of parameter settings on model performance, while visual examples reveal the interpretability of the model. These results validate the effectiveness and practicality of CSAM in handling visual and language multimodal tasks.