Large Language Models (LLMs) have rapidly advanced the field of natural language processing, enabling applications ranging from chatbots and virtual assistants to scientific research and education. Despite their impressive generative capabilities, a pressing concern remains: how trustworthy are these models when it comes to delivering accurate, contextually grounded information? This paper addresses this challenge by evaluating the trustworthiness of LLMs through a dual approach that leverages LlamaIndex for retrieval-augmented generation (RAG) and RAGAS for systematic performance evaluation. We investigate three LLMs – LLaMA 3 (Groq), Mistral-7B-Instruct (Together.ai), and GPT-4o-mini (OpenAI) – and compare their ability to retrieve, interpret, and generate accurate answers grounded in external data. Using a subset of the neural-bridge/rag-dataset-12000, we conduct a detailed, metric-driven evaluation. The results demonstrate that while GPT-4o-mini leads in most trustworthiness dimensions, LLaMA 3 exhibits strong competitiveness as an open-weight alternative. Mistral, though promising, falls short in certain key areas of contextual grounding and correctness.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Assessing the Trustworthiness of Large Language Models: A Two-Stage Framework Using Ragas and LlamaIndex

  • Alin-Gabriel Văduva,
  • Anca-Ioana Andreescu,
  • Simona-Vasilica Oprea

摘要

Large Language Models (LLMs) have rapidly advanced the field of natural language processing, enabling applications ranging from chatbots and virtual assistants to scientific research and education. Despite their impressive generative capabilities, a pressing concern remains: how trustworthy are these models when it comes to delivering accurate, contextually grounded information? This paper addresses this challenge by evaluating the trustworthiness of LLMs through a dual approach that leverages LlamaIndex for retrieval-augmented generation (RAG) and RAGAS for systematic performance evaluation. We investigate three LLMs – LLaMA 3 (Groq), Mistral-7B-Instruct (Together.ai), and GPT-4o-mini (OpenAI) – and compare their ability to retrieve, interpret, and generate accurate answers grounded in external data. Using a subset of the neural-bridge/rag-dataset-12000, we conduct a detailed, metric-driven evaluation. The results demonstrate that while GPT-4o-mini leads in most trustworthiness dimensions, LLaMA 3 exhibits strong competitiveness as an open-weight alternative. Mistral, though promising, falls short in certain key areas of contextual grounding and correctness.