In the task of video moment localization and highlight segment detection, it is crucial to understand the correlation between the video content and the query text because the video usually contains dynamic temporal information while the text provides the semantic and contextual descriptions. The effective fusion of the two can help the model capture the connection between events and the complete context more accurately. However, some existing methods tend to ignore this contextual integration, usually adopting simple feature splicing or global average pooling for information fusion, which cannot fully utilize the temporal characteristics in video and semantic details in text. In addition, some methods lack a flexible cross-modal alignment strategy when processing video and text, making the potential associations between the two not captured effectively. In order to solve the above problems, this paper proposes a conditional query-based algorithm for video moment localization and highlight segment detection (Conditional Query Model, CQM). The algorithm designs a conditional query decoder that enhances the performance of the conditional cross-attention mechanism by introducing conditional location query. Specifically, CQM introduces conditional location query during the decoding process, which enables each cross-attention head to focus on specific time intervals in the video clip, such as the start moment of a key event, the end moment, or a salient region within the clip.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CQM: Algorithm for Video Moment Localization and Highlight Segment Detection Based on Conditional Query

  • Yude Wang,
  • Xinyu Wang,
  • Fei Song,
  • Chuanxin Liu,
  • Zongqiang Liu

摘要

In the task of video moment localization and highlight segment detection, it is crucial to understand the correlation between the video content and the query text because the video usually contains dynamic temporal information while the text provides the semantic and contextual descriptions. The effective fusion of the two can help the model capture the connection between events and the complete context more accurately. However, some existing methods tend to ignore this contextual integration, usually adopting simple feature splicing or global average pooling for information fusion, which cannot fully utilize the temporal characteristics in video and semantic details in text. In addition, some methods lack a flexible cross-modal alignment strategy when processing video and text, making the potential associations between the two not captured effectively. In order to solve the above problems, this paper proposes a conditional query-based algorithm for video moment localization and highlight segment detection (Conditional Query Model, CQM). The algorithm designs a conditional query decoder that enhances the performance of the conditional cross-attention mechanism by introducing conditional location query. Specifically, CQM introduces conditional location query during the decoding process, which enables each cross-attention head to focus on specific time intervals in the video clip, such as the start moment of a key event, the end moment, or a salient region within the clip.