Scholarlink

<p>Retrieval-Augmented Generation (RAG) systems have gained significant attention for their ability to enhance language models by incorporating external knowledge sources. However, evaluating the effectiveness of these systems remains a challenge, as both the retrieval and generation components must be assessed independently and in combination. This study aims to provide a comprehensive evaluation of open-source large language models (LLMs) within a RAG system, using a dataset derived from diploma theses abstracts. To systematically assess performance, we generated a benchmark dataset consisting of 122 questions with reference answers, categorized into reasoning, fact-based, and summary-type questions. Using Elasticsearch as the retriever and several open-source LLMs as generators, we evaluated the system using the Ragas (Retrieval Augmented Generation Assessment) framework, focusing on retrieval effectiveness and answer quality. Our findings highlight the strengths and weaknesses of different models and retrieval strategies, offering insights into optimizing RAG implementations for academic and structured knowledge tasks. Our dataset and the corresponding evaluation code are publicly accessible at <a href="https://github.com/margitantal68/rag_paper">https://github.com/margitantal68/rag_paper</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating Open-Source LLMs in RAG Systems: A Benchmark on Diploma Theses Abstracts Using Ragas

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年7月16日

查看全文
  • Margit Antal,
  • Krisztian Buza

摘要

Retrieval-Augmented Generation (RAG) systems have gained significant attention for their ability to enhance language models by incorporating external knowledge sources. However, evaluating the effectiveness of these systems remains a challenge, as both the retrieval and generation components must be assessed independently and in combination. This study aims to provide a comprehensive evaluation of open-source large language models (LLMs) within a RAG system, using a dataset derived from diploma theses abstracts. To systematically assess performance, we generated a benchmark dataset consisting of 122 questions with reference answers, categorized into reasoning, fact-based, and summary-type questions. Using Elasticsearch as the retriever and several open-source LLMs as generators, we evaluated the system using the Ragas (Retrieval Augmented Generation Assessment) framework, focusing on retrieval effectiveness and answer quality. Our findings highlight the strengths and weaknesses of different models and retrieval strategies, offering insights into optimizing RAG implementations for academic and structured knowledge tasks. Our dataset and the corresponding evaluation code are publicly accessible at https://github.com/margitantal68/rag_paper.