Scholarlink

Generative AI powered by Large Language Model (LLM) can produce creative content including programming languages but is constrained by the training data. On the other hand, Question Answering Systems (QASs) are not limited by data biases or quality, but cannot detect human errors or ambiguities. Hence, integrating Generative AI in QASs can transform the functionality and user experience for better. Our work presents a comprehensive evaluation of the performance of four prominent Large Language Models (LLMs)–ChatGPT, Claude, Gemini and Llama3 - on the task of converting Natural Language Questions (NLQs) to SPARQL queries. We created a novel sample dataset by merging LC-QuAD 2.0 and QALD 10 datasets to ensure a diverse representation of question types, knowledge domains, and complexity levels. We evaluated the performance of each LLM and conducted an in-depth error analysis to understand capabilities and identify weaknesses for NLQ-to-SPARQL conversion, which can guide future research and development in this exciting field.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Enhancing Question Answering Systems with Generative AI: A Study of LLM Performance and Error Analysis

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Faiza Nuzhat,
  • Kanchan Shivashankar,
  • Nadine Steinmetz

摘要

Generative AI powered by Large Language Model (LLM) can produce creative content including programming languages but is constrained by the training data. On the other hand, Question Answering Systems (QASs) are not limited by data biases or quality, but cannot detect human errors or ambiguities. Hence, integrating Generative AI in QASs can transform the functionality and user experience for better. Our work presents a comprehensive evaluation of the performance of four prominent Large Language Models (LLMs)–ChatGPT, Claude, Gemini and Llama3 - on the task of converting Natural Language Questions (NLQs) to SPARQL queries. We created a novel sample dataset by merging LC-QuAD 2.0 and QALD 10 datasets to ensure a diverse representation of question types, knowledge domains, and complexity levels. We evaluated the performance of each LLM and conducted an in-depth error analysis to understand capabilities and identify weaknesses for NLQ-to-SPARQL conversion, which can guide future research and development in this exciting field.