Scholarlink

<p>The increasing prevalence and volume of short-form video content and real-time multimedia has made live video captioning indispensable. Consequently, there is a rising demand for autonomous systems that can understand the observed scene and describe it for applications ranging from large-scale content-indexing to live-commentary systems. Effective scene understanding requires precise object identification along with contextual interpretation of object interactions. This research achieves this through a novel feature fusion approach, combining low-level VGG16 CNN-extracted spatial features with YOLOv8 object detection results. The fused features are then processed through an LSTM-based encoder-decoder architecture, augmented with a customised category-aware attention mechanism, to enable real-time caption generation. Trained on the MSVD dataset, the proposed approach outperforms our baseline model by over 20% in BLEU-4 and METEOR scores, with a 54% drop in average inference time. Indeed, it is substantial to note that the motivation of this paper lies in improving foundational uni-modal methods for converting live video into natural language in rapid, resource-constrained environments, hypothesizing that advancements in this area can subsequently benefit the development of more efficient architectures leveraging lighter and more modern models.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Real-time video captioning using feature fusion and attention mechanism

  • 期刊论文

  • 发表日期：

    2025年9月5日

查看全文
  • K R Sarath Chandran,
  • Dejah Madhusankar,
  • Krithika Swaminathan

摘要

The increasing prevalence and volume of short-form video content and real-time multimedia has made live video captioning indispensable. Consequently, there is a rising demand for autonomous systems that can understand the observed scene and describe it for applications ranging from large-scale content-indexing to live-commentary systems. Effective scene understanding requires precise object identification along with contextual interpretation of object interactions. This research achieves this through a novel feature fusion approach, combining low-level VGG16 CNN-extracted spatial features with YOLOv8 object detection results. The fused features are then processed through an LSTM-based encoder-decoder architecture, augmented with a customised category-aware attention mechanism, to enable real-time caption generation. Trained on the MSVD dataset, the proposed approach outperforms our baseline model by over 20% in BLEU-4 and METEOR scores, with a 54% drop in average inference time. Indeed, it is substantial to note that the motivation of this paper lies in improving foundational uni-modal methods for converting live video into natural language in rapid, resource-constrained environments, hypothesizing that advancements in this area can subsequently benefit the development of more efficient architectures leveraging lighter and more modern models.