<p>In multimodal knowledge graph completion, information across different modalities (such as text and images) exist typically the semantic gap among distinct feature spaces. Many knowledge graph completion methods overemphasize the prediction of missing links in multimodal entities while neglecting the inductive reasoning ability for emerging entities. To overcome the above challenges, this paper introduces a novel modality and semantic alignments based neighborhood node retrieval approach, designed to address the inductive multimodal knowledge graph completion problem for emerging entities. Specifically, we propose a unified multimodal alignment module that captures the relevance of entities’ modality information within a shared representation space and strengthens the fusion representations between the query and target entities. This module filters the influence of visual noise context by comparing the similarity of images. Furthermore, the proposed multimodal fusion module facilitates hierarchical fusion of visual information, while leveraging contrastive learning to further enhance the semantic relevance between the query entity and the target entity. In the test prediction stage, we still use the semantic neighbor similarity and integrate it with the similarity distribution of the query entity to enhance the prediction accuracy of the result. Through extensive empirical evaluation on multiple real-world datasets, the proposed method shows significant effectiveness and robustness.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Modality and semantic alignments based neighborhood node retrieval for multimodal knowledge graph completion

  • Dabao Zhang,
  • Boyue Wang,
  • Lan Zhao,
  • Yongli Hu,
  • Baocai Yin

摘要

In multimodal knowledge graph completion, information across different modalities (such as text and images) exist typically the semantic gap among distinct feature spaces. Many knowledge graph completion methods overemphasize the prediction of missing links in multimodal entities while neglecting the inductive reasoning ability for emerging entities. To overcome the above challenges, this paper introduces a novel modality and semantic alignments based neighborhood node retrieval approach, designed to address the inductive multimodal knowledge graph completion problem for emerging entities. Specifically, we propose a unified multimodal alignment module that captures the relevance of entities’ modality information within a shared representation space and strengthens the fusion representations between the query and target entities. This module filters the influence of visual noise context by comparing the similarity of images. Furthermore, the proposed multimodal fusion module facilitates hierarchical fusion of visual information, while leveraging contrastive learning to further enhance the semantic relevance between the query entity and the target entity. In the test prediction stage, we still use the semantic neighbor similarity and integrate it with the similarity distribution of the query entity to enhance the prediction accuracy of the result. Through extensive empirical evaluation on multiple real-world datasets, the proposed method shows significant effectiveness and robustness.