Scholarlink

Question Answering is one of the prime applications of LLMs and other such transformer-based language models, which have been shown to achieve results as of yet unmatched in many NLP tasks. And yet, the automatic evaluation metrics used in benchmarking these systems are unsatisfactory, producing results that don’t correlate well to human judgement and offer no interpretation of the model’s failings. This paper introduces Q3AE, an evaluation method for QA systems that leverages adversarial attacks to measure the robustness of answerer models when tested against challenging input. We test the method by introducing spelling mistakes and other such keyboarding errors, common in day to day use, into the SQuAD dataset, translated to Portuguese, and use it as input against LLMs of the LLaMA and Gemma architectures, of different sizes. We find that these models are surprisingly sensitive to even the tiniest alterations, responding poorly to modified examples that a human being could still easily understand.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating Question Answering Model Robustness Against Typographical Mistakes Using Adversarial Attacks

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Ricardo Saraiva Grava,
  • Anarosa Alves Franco Brandão,
  • Sarajane Marques Peres,
  • Fabio Gagliardi Cozman

摘要

Question Answering is one of the prime applications of LLMs and other such transformer-based language models, which have been shown to achieve results as of yet unmatched in many NLP tasks. And yet, the automatic evaluation metrics used in benchmarking these systems are unsatisfactory, producing results that don’t correlate well to human judgement and offer no interpretation of the model’s failings. This paper introduces Q3AE, an evaluation method for QA systems that leverages adversarial attacks to measure the robustness of answerer models when tested against challenging input. We test the method by introducing spelling mistakes and other such keyboarding errors, common in day to day use, into the SQuAD dataset, translated to Portuguese, and use it as input against LLMs of the LLaMA and Gemma architectures, of different sizes. We find that these models are surprisingly sensitive to even the tiniest alterations, responding poorly to modified examples that a human being could still easily understand.