Scholarlink

<p>This paper presents a multilingual benchmark of Retrieval-Augmented Generation (RAG) systems applied to Arabic and English documents. We evaluate five open-source Large Language Models (LLMs): Llama3.3, Llama3.1, Qwen2.5, Command R7b, and DeepSeek R1, across four knowledge domains: History, Law, Management, and Science. Our hybrid evaluation framework integrates standard lexical metrics (Precision, Recall, F1-Score) with an LLM-based semantic assessment using an LLM-based evaluator to simulate human judgment of answer correctness. Results reveal a consistent performance advantage for English across all models’ evaluations and underscore the limitations of purely lexical metrics for morphologically rich languages like Arabic. Llama3.3 achieved the highest overall semantic accuracy (LLM-based scores reaching approximately 0.78 in English and 0.70 in Arabic), compared to substantially lower Arabic semantic scores for DeepSeek R1 (as low as approximately 0.09), albeit with greater latency (averaging 40-68&#xa0;s compared to 3-4&#xa0;s for lighter models). Command R7b provided a balanced trade-off between model size and performance, making it a good candidate for resource-constrained devices, such as edge devices. Qwen2.5 and Llama3.1 provided a balanced trade-off between speed and accuracy. DeepSeek R1 showed the lowest scores, particularly in Arabic, and occasionally defaulted to Chinese responses, empirically supporting the latent language hypothesis by revealing an internal preference for training-dominant, high-resource languages. These results further support the latent language hypothesis by suggesting an internal model bias toward high-resource languages. Finally, beyond the 143% precision gain from prompt engineering, this paper contributes an important practical reproducible framework for multilingual RAG benchmarking and insights for optimizing performance on resource-constrained devices.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Benchmarking and AI-assisted human-like evaluation of retrieval-augmented generation for Arabic and English documents

  • 期刊论文

  • 发表日期：

    2026年7月31日

查看全文
  • Bassam J. Mohd,
  • Khalil M. Ahmad Yousef,
  • Salah G. Abu Ghalyon

摘要

This paper presents a multilingual benchmark of Retrieval-Augmented Generation (RAG) systems applied to Arabic and English documents. We evaluate five open-source Large Language Models (LLMs): Llama3.3, Llama3.1, Qwen2.5, Command R7b, and DeepSeek R1, across four knowledge domains: History, Law, Management, and Science. Our hybrid evaluation framework integrates standard lexical metrics (Precision, Recall, F1-Score) with an LLM-based semantic assessment using an LLM-based evaluator to simulate human judgment of answer correctness. Results reveal a consistent performance advantage for English across all models’ evaluations and underscore the limitations of purely lexical metrics for morphologically rich languages like Arabic. Llama3.3 achieved the highest overall semantic accuracy (LLM-based scores reaching approximately 0.78 in English and 0.70 in Arabic), compared to substantially lower Arabic semantic scores for DeepSeek R1 (as low as approximately 0.09), albeit with greater latency (averaging 40-68 s compared to 3-4 s for lighter models). Command R7b provided a balanced trade-off between model size and performance, making it a good candidate for resource-constrained devices, such as edge devices. Qwen2.5 and Llama3.1 provided a balanced trade-off between speed and accuracy. DeepSeek R1 showed the lowest scores, particularly in Arabic, and occasionally defaulted to Chinese responses, empirically supporting the latent language hypothesis by revealing an internal preference for training-dominant, high-resource languages. These results further support the latent language hypothesis by suggesting an internal model bias toward high-resource languages. Finally, beyond the 143% precision gain from prompt engineering, this paper contributes an important practical reproducible framework for multilingual RAG benchmarking and insights for optimizing performance on resource-constrained devices.