Scholarlink

This study assesses the performance of two large language models (ChatGPT 4.5 and Gemini Advanced) in responding to questions about archiving standards. Results show a low accuracy rate, with only 29% of answers fully correct, and 71% being partial or approximate. The findings highlight that answer reliability depends on question type (closed questions yield more accurate and consistent responses) and formulation. Well-structured prompts, such as zero-shot prompting, improved performance. Overall, ChatGPT 4.5 slightly outperformed Gemini Advanced in accuracy and instruction-following, with response rates of 70% and 65% respectively. The study recommends further exploration of prompting strategies like few-shot and chain-of-thought to support archivists in identifying relevant standards.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Assessment LLM’s Capabilities in Archiving Standards Research: An Evaluation Study

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • El Mokhtar Hani,
  • Bouchra El Idrissi

摘要

This study assesses the performance of two large language models (ChatGPT 4.5 and Gemini Advanced) in responding to questions about archiving standards. Results show a low accuracy rate, with only 29% of answers fully correct, and 71% being partial or approximate. The findings highlight that answer reliability depends on question type (closed questions yield more accurate and consistent responses) and formulation. Well-structured prompts, such as zero-shot prompting, improved performance. Overall, ChatGPT 4.5 slightly outperformed Gemini Advanced in accuracy and instruction-following, with response rates of 70% and 65% respectively. The study recommends further exploration of prompting strategies like few-shot and chain-of-thought to support archivists in identifying relevant standards.