Scholarlink

<p>Transformer and Long Short-Term Memory (LSTM) have been extensively employed in video captioning, since they can cope with the temporal dependencies within both video frames and the corresponding descriptions. However, in traditional Transformer or LSTM models, the currently generated hidden state cannot establish dependencies with subsequent hidden states carrying underly assumed semantics. Consequently, the predicted word is only associated with the preceding hidden states, disregarding the overall context. This leads to imbalanced outputs, where captions frequently have satisfactory prefixes but unsatisfactory suffixes. Inspired by the cognitive behavior of humans considering the consistency between local words and global semantics when describing videos, we propose two unique schemes on different architectures: the Transformer-based Linguistic Deliberation Cascade Transformer (LDCT) and the LSTM-based Linguistic Deliberation Cascade LSTM (LDCL). Our method is advantageous in two aspects: (1) We introduce the Linguistic Deliberation Modules (LDM), which adaptively constructs global semantic information based on the semantic relevance matrix calculated for the entire sentence. By effectively integrating this global semantic information, LDM empowers the current word to comprehend the complete contextual information and mitigates the issue of imbalanced outputs. (2) Through the design of several fusion methods tailored to different architectures (LDCT and LDCL) and the regularization of target sequences generated by the visual-language interaction module during the end-to-end training process, our method successfully enhances overall performance and structural generalization ability. We conduct extensive experiments on the widely used datasets MSVD and MSR-VTT to demonstrate the effectiveness and generalization capabilities of our method in comparison to state-of-the-art approaches.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Refined linguistic deliberation for video captioning via cascade transformer and LSTM

  • 期刊论文

  • 发表日期：

    2025年4月21日

查看全文
  • Shuqin Chen,
  • Zhixin Sun,
  • Li Yang,
  • Yikang Hu,
  • Shifeng Wu,
  • Yi Zhang

摘要

Transformer and Long Short-Term Memory (LSTM) have been extensively employed in video captioning, since they can cope with the temporal dependencies within both video frames and the corresponding descriptions. However, in traditional Transformer or LSTM models, the currently generated hidden state cannot establish dependencies with subsequent hidden states carrying underly assumed semantics. Consequently, the predicted word is only associated with the preceding hidden states, disregarding the overall context. This leads to imbalanced outputs, where captions frequently have satisfactory prefixes but unsatisfactory suffixes. Inspired by the cognitive behavior of humans considering the consistency between local words and global semantics when describing videos, we propose two unique schemes on different architectures: the Transformer-based Linguistic Deliberation Cascade Transformer (LDCT) and the LSTM-based Linguistic Deliberation Cascade LSTM (LDCL). Our method is advantageous in two aspects: (1) We introduce the Linguistic Deliberation Modules (LDM), which adaptively constructs global semantic information based on the semantic relevance matrix calculated for the entire sentence. By effectively integrating this global semantic information, LDM empowers the current word to comprehend the complete contextual information and mitigates the issue of imbalanced outputs. (2) Through the design of several fusion methods tailored to different architectures (LDCT and LDCL) and the regularization of target sequences generated by the visual-language interaction module during the end-to-end training process, our method successfully enhances overall performance and structural generalization ability. We conduct extensive experiments on the widely used datasets MSVD and MSR-VTT to demonstrate the effectiveness and generalization capabilities of our method in comparison to state-of-the-art approaches.