Multi-modal knowledge graphs (MMKGs) present distinct advantages over unimodal knowledge graphs by leveraging diverse modal attributes for more comprehensive entity descriptions. Most existing studies on multi-modal knowledge graph completion focus on link prediction and numerical attribute completion, neglecting another significant issue, i.e., the incompleteness of image attribute. To fill the gap, this paper formulates a novel task Image Attribute Completion, and develops a dedicated model entitled MAGIC (Multi-modal Assisted imaGe attrIbutes Completion). Specifically, the model consists of two main components: (1) The module EEM is designed to Encode Entity with Different Modalities (i.e., structural information, discrete attributes, text, and images) by employing distinct encoders to learn modality-specific embeddings. Then, these representations are dynamically fused by calculating the entropy of each modality. (2) Based on the fused embeddings of entities, the module EIA is introduced to perform Entity-Image Alignment, which is achieved through a two-step process. First, we calculate the similarity between entities and candidate images. Second, two loss functions are combined to simultaneously enhance modality alignment and classification accuracy. The experiments conducted on three real-world datasets demonstrate the superiority of our proposed model.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Image Attribute Completion: A Novel Task for Multi-modal Knowledge Graph

  • Yirui Ma,
  • Qian Zhou,
  • Wei Chen,
  • Xi Chen,
  • Xiaofang Zhang,
  • Lei Zhao

摘要

Multi-modal knowledge graphs (MMKGs) present distinct advantages over unimodal knowledge graphs by leveraging diverse modal attributes for more comprehensive entity descriptions. Most existing studies on multi-modal knowledge graph completion focus on link prediction and numerical attribute completion, neglecting another significant issue, i.e., the incompleteness of image attribute. To fill the gap, this paper formulates a novel task Image Attribute Completion, and develops a dedicated model entitled MAGIC (Multi-modal Assisted imaGe attrIbutes Completion). Specifically, the model consists of two main components: (1) The module EEM is designed to Encode Entity with Different Modalities (i.e., structural information, discrete attributes, text, and images) by employing distinct encoders to learn modality-specific embeddings. Then, these representations are dynamically fused by calculating the entropy of each modality. (2) Based on the fused embeddings of entities, the module EIA is introduced to perform Entity-Image Alignment, which is achieved through a two-step process. First, we calculate the similarity between entities and candidate images. Second, two loss functions are combined to simultaneously enhance modality alignment and classification accuracy. The experiments conducted on three real-world datasets demonstrate the superiority of our proposed model.