Short-response grading is a central issue in reading comprehension evaluation. In this regard, this work first introduces an enriched dataset with both human-written and artificial intelligence–generated responses, and then compares transformer-based models, similarity metrics, and large language models for automatic validation of open-ended responses. The fine-tuned RoBERTa binary classifier achieved competitive performance, but DeepSeek-V3 outperformed all models, including ChatGPT-4o. A Sentence Transformer model trained with contrastive learning showed limitations in detecting incorrect answers. We discuss the strengths and weaknesses of each approach and propose hybrid models that are better aligned with pedagogical goals.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Validation of Short Answers for Open Questions from the FairytaleQA-Sp Corpus

  • Margarita Aviña-Corral,
  • Delia Irazú Hernández-Farías,
  • Aurelio López-López,
  • Manuel Montes-y-Gómez,
  • Luis Villaseñor-Pineda

摘要

Short-response grading is a central issue in reading comprehension evaluation. In this regard, this work first introduces an enriched dataset with both human-written and artificial intelligence–generated responses, and then compares transformer-based models, similarity metrics, and large language models for automatic validation of open-ended responses. The fine-tuned RoBERTa binary classifier achieved competitive performance, but DeepSeek-V3 outperformed all models, including ChatGPT-4o. A Sentence Transformer model trained with contrastive learning showed limitations in detecting incorrect answers. We discuss the strengths and weaknesses of each approach and propose hybrid models that are better aligned with pedagogical goals.