Scholarlink

<p>Semantic similarity is a fundamental task in natural language processing. Despite the impressive performances achieved by the existing knowledge enhancement methods in semantic similarity, they predominantly target at the unimodal information or depend solely on knowledge derived from a singular source. How to deal with the refined knowledge enhancement while well considering the complement between multimodal information is still a challenging problem, especially in the era of rapidly evolving large language models (LLMs), which offer new opportunities for knowledge enrichment and reasoning. To address this, we focus on semantic similarity representation in Chinese and introduce MuS<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(^2\)</EquationSource> </InlineEquation>, a novel Multi-Source knowledge enhancement for Multimodal <Emphasis Type="Underline">S</Emphasis>emantic representation model, which integrates generative textual representations with retrieval-based visual features. By leveraging LLMs to enrich retrieved content and employing retrieval mechanisms to counteract LLM-induced hallucinations, MuS<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(^2\)</EquationSource> </InlineEquation> achieves a synergistic balance between generation and retrieval. Specifically, the MuS<InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(^2\)</EquationSource> </InlineEquation> model comprises four core components: (1) A multi-source knowledge enhancement module that leverages LLMs for semantic textual enrichment and integrates image knowledge through search engine retrieval; (2) A dual-stream multimodal encoder processing textual and image modalities separately; (3) A multimodal fusion module aligning and combining the encoded representations; and (4) A semantic similarity calculation module generating pairwise lexical similarity metrics. Extensive experiments on four public datasets demonstrate the effectiveness of MuS<InlineEquation ID="IEq4"> <EquationSource Format="TEX">\(^2\)</EquationSource> </InlineEquation>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multi-source knowledge enhancement for multimodal semantic representation

  • 期刊论文

  • 发表日期：

    2026年2月2日

查看全文
  • Pei-Yuan Lai,
  • Qing-Yun Dai,
  • De-Zhang Liao,
  • Huan-Tao Cai,
  • Man-Sheng Chen,
  • Chang-Dong Wang

摘要

Semantic similarity is a fundamental task in natural language processing. Despite the impressive performances achieved by the existing knowledge enhancement methods in semantic similarity, they predominantly target at the unimodal information or depend solely on knowledge derived from a singular source. How to deal with the refined knowledge enhancement while well considering the complement between multimodal information is still a challenging problem, especially in the era of rapidly evolving large language models (LLMs), which offer new opportunities for knowledge enrichment and reasoning. To address this, we focus on semantic similarity representation in Chinese and introduce MuS \(^2\) , a novel Multi-Source knowledge enhancement for Multimodal Semantic representation model, which integrates generative textual representations with retrieval-based visual features. By leveraging LLMs to enrich retrieved content and employing retrieval mechanisms to counteract LLM-induced hallucinations, MuS \(^2\) achieves a synergistic balance between generation and retrieval. Specifically, the MuS \(^2\) model comprises four core components: (1) A multi-source knowledge enhancement module that leverages LLMs for semantic textual enrichment and integrates image knowledge through search engine retrieval; (2) A dual-stream multimodal encoder processing textual and image modalities separately; (3) A multimodal fusion module aligning and combining the encoded representations; and (4) A semantic similarity calculation module generating pairwise lexical similarity metrics. Extensive experiments on four public datasets demonstrate the effectiveness of MuS \(^2\) .