Collaborative Multimodal Learning for Human-Human Interaction Recognition in Videos
摘要
Recognizing the interaction of people is a basic but challenging task in video understanding. In this paper, we propose a new task in this field, named Human-Human Interaction Recognition (HHIR). Our research focuses on recognizing interactions between two individuals in a scene, particularly those involving communication. It is different from the traditional action classification task. HHIR requires modeling complex and multimodal contextual cues across extended video scenes. To address this challenge, we propose a collaborative multimodal learning framework that integrates visual, textual, and audio modalities through an enhanced cross-attention mechanism. Our method leverages ViViT model for spatiotemporal visual feature extraction instead of traditional CNN models, BERT for dialogue understanding, and AST for audio feature extraction. Furthermore, we incorporate a collaborative knowledge integration strategy to enhance cross-modal interactions and apply class weighting to mitigate data imbalance issues. We build a new dataset, constructed from 12 diverse movies, capturing interactions between two individuals instead of isolated actions. Experimental results on this dataset demonstrate that our model achieves 78.54% Micro-F1 and 62.47% Macro-F1, outperforming state-of-the-art methods in multimodal human interaction recognition. This demonstrates the effectiveness of collaborative multimodal learning in capturing complex human interactions. The source code is available at https://github.com/hvngocthanh-CS/Human-Human-Interaction-Recognition-in-videos.git