KITE-MRE: Knowledge-Infused and Transformer-Enhanced Multimodal Relation Extraction
摘要
Multimodal Relation Extraction (MRE) is a task that identifies semantic relations between two entities from image-text pairs. Existing methods often suffer from insufficient extraction accuracy and low cross-modal fusion efficiency due to inadequate auxiliary information. In this paper, we propose a multimodal relation extraction framework KITE-MRE,which integrating large model generation and knowledge graph (KG) to provide abundant additional auxiliary information. Specifically, prompts guide large language models to construct image semantic entity triplets and generate image graphs, while knowledge graph entity linking technology is applied in text graphs to build a unified graph structure. Additionally, we introduce the Graph Information Bottleneck (GIB) technique combined with Graph Convolutional Networks (GCN) to optimize, and perform feature fusion by integrating semantic and structural multi-perspective information to predict relations. Experimental results demonstrate that this approach provides a new path for multimodal data processing, effectively addressing issues of knowledge graph incompleteness and ambiguous relation generation by large models in cross-modal scenarios.