Multi-granularity visual relationship reasoning with heterogeneous graph interaction for video question answering
摘要
Video question answering requires the effective utilization of multimodal features in videos as well as reasoning over complex relationships among them. However, existing video question answering models often overlook the relationship information at different granularities and struggle to effectively fuse multimodal features relevant to the question. To address these issues, this paper proposes a multi-granularity visual relationship reasoning with heterogeneous graph interaction model. Specifically, we construct global graphs and object graphs based on word-aware global and object features to capture coarse-grained and fine-grained visual relationships, respectively. A caption graph is further introduced to model textual relationships among word features. Moreover, to leverage the complementary information from multimodal features in videos, we design a question-related heterogeneous graph interaction module to embed question-related multi-granularity visual features and caption features into a unified heterogeneous graph. A graph attention network is used to facilitate information interaction among features of different granularities and modalities. Extensive experiments on two public datasets demonstrate the effectiveness of the proposed model.