Scholarlink

Question Answering (QA) is a widely used benchmark to evaluate the reasoning and comprehension abilities of large language models (LLMs). In this study, the performance of five pre-trained QA models, trained on the SQuAD dataset, is evaluated across varying input text lengths, categorized as small (500–1000 words), medium (1000–2000 words), and large (2000–4000 words). All models are used without any fine-tuning or additional training. A retrieval-based approach is employed to extract the most relevant context from lengthy documents. The model outputs are assessed using a range of evaluation metrics, including precision, recall, F1 score (to capture exactness), BLEU (fluency), ROUGE-1, ROUGE-2, and ROUGE-L. Among the models evaluated, albert-xxlarge-v2-squad2 demonstrated the best overall performance, followed by roberta-base-squad2. Notably, the highest precision achieved was 0.822, while the ROUGE-1 and ROUGE-L scores reached 0.802. In addition, a 39.6% improvement in the average ROUGE scores was observed on a strong baseline, based on evaluations conducted on a custom dataset of 50 different texts. Despite these results, the models exhibited relatively lower scores in other metrics, suggesting room for further optimization. This study considers inputs of up to 4000 words. Future work will explore longer inputs and domain-specific datasets.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating Question Answering Performance of SQuAD-Based LLMs Across Varying Text Lengths

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Swati Raj,
  • Saurabh Srivastava

摘要

Question Answering (QA) is a widely used benchmark to evaluate the reasoning and comprehension abilities of large language models (LLMs). In this study, the performance of five pre-trained QA models, trained on the SQuAD dataset, is evaluated across varying input text lengths, categorized as small (500–1000 words), medium (1000–2000 words), and large (2000–4000 words). All models are used without any fine-tuning or additional training. A retrieval-based approach is employed to extract the most relevant context from lengthy documents. The model outputs are assessed using a range of evaluation metrics, including precision, recall, F1 score (to capture exactness), BLEU (fluency), ROUGE-1, ROUGE-2, and ROUGE-L. Among the models evaluated, albert-xxlarge-v2-squad2 demonstrated the best overall performance, followed by roberta-base-squad2. Notably, the highest precision achieved was 0.822, while the ROUGE-1 and ROUGE-L scores reached 0.802. In addition, a 39.6% improvement in the average ROUGE scores was observed on a strong baseline, based on evaluations conducted on a custom dataset of 50 different texts. Despite these results, the models exhibited relatively lower scores in other metrics, suggesting room for further optimization. This study considers inputs of up to 4000 words. Future work will explore longer inputs and domain-specific datasets.