Scholarlink

<p>Understanding and analyzing video actions is crucial for generating context-aware descriptions in applications such as intelligent monitoring and autonomous systems. This work proposes a novel video captioning framework that integrates visual and textual modalities to produce natural language descriptions. Visual features are extracted from video frames using ResNet50 and converted into patch embeddings, which are processed through a GPT-2–based encoder–decoder model. To enhance alignment between modalities, the framework employs multi-head self-attention and cross-attention mechanisms. The model is evaluated on the Microsoft Research Video Description Corpus (MSVD) and the Berkeley DeepDrive eXplanation (BDD-X) datasets using BLEU, CIDEr, METEOR, and ROUGE-L metrics. Experimental results demonstrate superior performance compared to traditional methods, achieving BLEU-4 scores of 0.778 (MSVD) and 0.755 (BDD-X), CIDEr scores of 1.315 (MSVD) and 1.235 (BDD-X), METEOR scores of 0.329 (MSVD) and 0.312 (BDD-X), and ROUGE-L scores of 0.795 (MSVD) and 0.782 (BDD-X). By generating human-like, contextually relevant captions and improving interpretability, this research contributes to advancing explainable AI for real-world video-based applications.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Towards explainable AI: multi-modal transformer for video-based image description generation

  • 期刊论文

  • 发表日期：

    2026年3月9日

查看全文
  • Lakshita Agarwal,
  • Bindu Verma

摘要

Understanding and analyzing video actions is crucial for generating context-aware descriptions in applications such as intelligent monitoring and autonomous systems. This work proposes a novel video captioning framework that integrates visual and textual modalities to produce natural language descriptions. Visual features are extracted from video frames using ResNet50 and converted into patch embeddings, which are processed through a GPT-2–based encoder–decoder model. To enhance alignment between modalities, the framework employs multi-head self-attention and cross-attention mechanisms. The model is evaluated on the Microsoft Research Video Description Corpus (MSVD) and the Berkeley DeepDrive eXplanation (BDD-X) datasets using BLEU, CIDEr, METEOR, and ROUGE-L metrics. Experimental results demonstrate superior performance compared to traditional methods, achieving BLEU-4 scores of 0.778 (MSVD) and 0.755 (BDD-X), CIDEr scores of 1.315 (MSVD) and 1.235 (BDD-X), METEOR scores of 0.329 (MSVD) and 0.312 (BDD-X), and ROUGE-L scores of 0.795 (MSVD) and 0.782 (BDD-X). By generating human-like, contextually relevant captions and improving interpretability, this research contributes to advancing explainable AI for real-world video-based applications.