Scholarlink

<p>Multimodal dense video captioning aims to temporally localize and describe events in untrimmed videos by leveraging massive heterogeneous high-dimensional spatiotemporal data streams, posing a significant computational challenge. The Enhanced Visual Multimodal Fusion (EVMF) framework is proposed, featuring a four-modality parallel multimodal processing architecture and three high-performance computing (HPC) innovations: (1) an Isomerous Fusion Encoder (IFE) with modality-aware attention mechanisms that resolve semantic disparities through learnable cross-modal alignment strategies, enabling integration of heterogeneous features; (2) a Visual Modality Enhancement Pipeline (VMEP) that employs dual parallel feature extraction models to address scene fragmentation by generating image captions and scene information; and (3) a multimodal LLM-based captioning module incorporating self-reflective contextual retrieval and adaptive evaluative mechanisms to autonomously refine outputs and reduce hallucinations. The model’s effectiveness is demonstrated through comprehensive evaluation on the ActivityNet Captions dataset, achieving state-of-the-art results in BLEU@N, METEOR, ROUGE-L, and CIDEr-D, while optimizing the trade-off between semantic accuracy and computational cost.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal fusion and knowledge enhancement for accurate video captioning

  • 期刊论文

  • 发表日期：

    2026年2月17日

查看全文
  • Ruizhe Zhong,
  • Qingchuan Zhang,
  • Haisheng Li,
  • Min Zuo

摘要

Multimodal dense video captioning aims to temporally localize and describe events in untrimmed videos by leveraging massive heterogeneous high-dimensional spatiotemporal data streams, posing a significant computational challenge. The Enhanced Visual Multimodal Fusion (EVMF) framework is proposed, featuring a four-modality parallel multimodal processing architecture and three high-performance computing (HPC) innovations: (1) an Isomerous Fusion Encoder (IFE) with modality-aware attention mechanisms that resolve semantic disparities through learnable cross-modal alignment strategies, enabling integration of heterogeneous features; (2) a Visual Modality Enhancement Pipeline (VMEP) that employs dual parallel feature extraction models to address scene fragmentation by generating image captions and scene information; and (3) a multimodal LLM-based captioning module incorporating self-reflective contextual retrieval and adaptive evaluative mechanisms to autonomously refine outputs and reduce hallucinations. The model’s effectiveness is demonstrated through comprehensive evaluation on the ActivityNet Captions dataset, achieving state-of-the-art results in BLEU@N, METEOR, ROUGE-L, and CIDEr-D, while optimizing the trade-off between semantic accuracy and computational cost.