Scholarlink

This study evaluates the semantic similarity performance of various embedding models when used in Retrieval Augmented Generation (RAG) systems powered by two recent large language models (LLMs): DeepSeek-R1-Distill-Qwen-1.5B and Gemma3-1B. We benchmarked 22 embedding models using the SQuAD 1.1 and HotPotQA datasets and assessed their output via the average BERT-Score Semantic Similarity (BSSS Avg). Our results show that Gemma3-1B consistently outperforms DeepSeek-R1, achieving a peak BSSS Avg of 0.98 with codebert-base on SQuAD 1.1 and 0.91 with instructor-base on HotPotQA. The study highlights that instruction-tuned and domain-specific embeddings significantly enhance performance, especially for complex, multi-hop reasoning tasks. These insights offer practical guidance on optimizing RAG pipelines through informed selection of embedding and LLM configurations.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Comparative Semantic Similarity for RAG: A Study of Embedding Models in Recent Large Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Vinh Dinh Nguyen,
  • Nhan Huu Tran,
  • Phong Van Nguyen,
  • Khoa Anh Dao,
  • Narayan C. Debnath

摘要

This study evaluates the semantic similarity performance of various embedding models when used in Retrieval Augmented Generation (RAG) systems powered by two recent large language models (LLMs): DeepSeek-R1-Distill-Qwen-1.5B and Gemma3-1B. We benchmarked 22 embedding models using the SQuAD 1.1 and HotPotQA datasets and assessed their output via the average BERT-Score Semantic Similarity (BSSS Avg). Our results show that Gemma3-1B consistently outperforms DeepSeek-R1, achieving a peak BSSS Avg of 0.98 with codebert-base on SQuAD 1.1 and 0.91 with instructor-base on HotPotQA. The study highlights that instruction-tuned and domain-specific embeddings significantly enhance performance, especially for complex, multi-hop reasoning tasks. These insights offer practical guidance on optimizing RAG pipelines through informed selection of embedding and LLM configurations.