Enhancing Subject-Oriented Video Captioning with Predicate-Guided Action Modeling
摘要
Subject-oriented video captioning generates natural language descriptions focusing on a specified subject’s activities. While prior works mainly utilize subject input to guide frame extraction and content encoding, they often neglect subject-environment interactions and detailed action modeling, leading to inaccuracies in action-related captions. To tackle this issue, we propose an enhanced framework that leverages predicate constraints and hierarchical interaction mechanisms to guide the model in understanding the actions of the subjects in the video, thereby enabling a more accurate summary of the subjects’ behaviors and generating more precise captions. Specifically, we first adopt predicates extracted from captions as supervision signal for learning subject-related action feature. Next, we utilize Swin-Transformer to extract hierarchical video features and facilitate hierarchical interactions between the subject features and the output of each layer, thereby capturing action semantics across various scales. To refine action feature learning, we introduce a composite loss function combining cosine similarity and mean squared error (MSE) to ensure alignment with the target embeddings. Finally, a gate mechanism dynamically fuses the video, subject, and action features, which are then input into a BERT-based generator to produce high-quality captions. Experimental results on the SO-MSVD and SO-MSRVTT datasets demonstrate that our approach significantly enhances captioning accuracy, particularly in terms of action verb precision, while maintaining computational efficiency suitable for real-world applications.