Machine Reading Comprehension (MRC) is a central benchmark for assessing whether language models can integrate comprehension and reasoning beyond surface-level pattern matching. While Large Language Models (LLMs) achieve strong performance on many NLP tasks, their robustness in reasoning-driven MRC remains an open question. This study provides a comparison of reasoning-oriented and general-purpose LLMs on two challenging benchmarks: SQuAD 2.0, which mixes answerable and unanswerable questions, and AdversarialQA, which introduces context-level perturbations. The results reveal two distinct performance profiles. General-purpose models, such as Mistral Medium and Pixtral Large Latest, exhibit moderate robustness to adversarial noise (average F1 ≈ 0.58) but achieve lower scores in rejecting unanswerable questions (F1 ≈ 0.25). Conversely, reasoning-oriented models, such as Gemini 2.5 Pro and Mistral Large Latest, attain higher F1 scores on unanswerable cases (≈ 0.85). These observations highlight a performance disparity between the two categories, suggesting that the ability to recognize when no valid answer exists is a more discriminative evaluation metric than extractive accuracy alone.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

The Role of Reasoning in LLMs: A Comparative Study on Machine Reading Comprehension Using the SQuAD 2.0 and AdversarialQA Datasets

  • Mahaman Sanoussi Yahaya Alassan,
  • El Hassane Ettifouri,
  • Walid Dahhane

摘要

Machine Reading Comprehension (MRC) is a central benchmark for assessing whether language models can integrate comprehension and reasoning beyond surface-level pattern matching. While Large Language Models (LLMs) achieve strong performance on many NLP tasks, their robustness in reasoning-driven MRC remains an open question. This study provides a comparison of reasoning-oriented and general-purpose LLMs on two challenging benchmarks: SQuAD 2.0, which mixes answerable and unanswerable questions, and AdversarialQA, which introduces context-level perturbations. The results reveal two distinct performance profiles. General-purpose models, such as Mistral Medium and Pixtral Large Latest, exhibit moderate robustness to adversarial noise (average F1 ≈ 0.58) but achieve lower scores in rejecting unanswerable questions (F1 ≈ 0.25). Conversely, reasoning-oriented models, such as Gemini 2.5 Pro and Mistral Large Latest, attain higher F1 scores on unanswerable cases (≈ 0.85). These observations highlight a performance disparity between the two categories, suggesting that the ability to recognize when no valid answer exists is a more discriminative evaluation metric than extractive accuracy alone.