A common-specific feature cross-fusion attention mechanism for KGVQA

Knowledge graph-based visual question answering aims to utilize the information in the knowledge graph to assist in answering complex questions that are difficult to answer based on image features alone. However, using knowledge graphs increases the difficulty of understanding facts by the model and introduces the possibility of generating noise, making it challenging to understand the facts and find answers. Previous multimodal fusion approaches typically treat the features of each modality as equally essential and implicitly explore the interactions between different modalities. However, we observe that when fusing text features with image features, image features can provide two types of information: common features and specific features. The common features can significantly enhance the text features and make the answer classification model more robust. In contrast, the specific features can complement the text features while providing different viewpoints, thus improving the performance of answer classification together with the common features. We propose a common-specific feature cross-fusion attention mechanism (CS-CFAN) approach based on these two observations. Unlike existing methods, CS-CFAN aims to learn how to extract and efficiently fuse features from complex multimodal data to solve complex questions that require external knowledge to assist in answering. On the F-VQA dataset, compared to the baseline model, our model achieves a 1.97% improvement with the same feature extraction method without a knowledge graph and reaches 82.66% accuracy with a knowledge graph.