<p>Image captioning aims to generate natural language descriptions for visual content by bridging computer vision and natural language processing. Recent large-scale visual-language pre-training models and Transformer-based architectures have significantly improved image captioning performance. However, the self-attention mechanism in Transformers struggles to balance modeling global dependencies with capturing local semantic continuity. Furthermore, existing methods commonly face the issue of supervision signals being biased toward a single modality during cross-modal training, resulting in insufficient visual-linguistic alignment. To address these limitations, we propose the Parallel Attention-MambaBlock module, which fuses the global context modeling capability of self-attention with the fine-grained sequence representation advantage of MambaBlock to achieve complementary feature learning. Simultaneously, we introduce a dual-teacher distillation framework that leverages visual and linguistic teachers to provide balanced and complementary supervision signals, comprehensively enhancing cross-modal representation capabilities. Finally, we develop an adaptive gated fusion mechanism that dynamically balances the weights of attention and MambaBlock outputs. This mechanism enables the model to adjust fusion weights based on contextual demands, efficiently integrating their optimized outputs. Extensive experiments on the MSCOCO benchmark demonstrate that our DAME-Cap model outperforms most state-of-the-art models.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DAME-Cap: a dual-teacher and parallel attention–mambablock framework for image captioning

  • Kangzhen He,
  • Juan Zhang,
  • Yongbin Gao,
  • Zhijun Fang

摘要

Image captioning aims to generate natural language descriptions for visual content by bridging computer vision and natural language processing. Recent large-scale visual-language pre-training models and Transformer-based architectures have significantly improved image captioning performance. However, the self-attention mechanism in Transformers struggles to balance modeling global dependencies with capturing local semantic continuity. Furthermore, existing methods commonly face the issue of supervision signals being biased toward a single modality during cross-modal training, resulting in insufficient visual-linguistic alignment. To address these limitations, we propose the Parallel Attention-MambaBlock module, which fuses the global context modeling capability of self-attention with the fine-grained sequence representation advantage of MambaBlock to achieve complementary feature learning. Simultaneously, we introduce a dual-teacher distillation framework that leverages visual and linguistic teachers to provide balanced and complementary supervision signals, comprehensively enhancing cross-modal representation capabilities. Finally, we develop an adaptive gated fusion mechanism that dynamically balances the weights of attention and MambaBlock outputs. This mechanism enables the model to adjust fusion weights based on contextual demands, efficiently integrating their optimized outputs. Extensive experiments on the MSCOCO benchmark demonstrate that our DAME-Cap model outperforms most state-of-the-art models.