Scholarlink

<p>High-stakes assessment is crucial for evaluating student performance and making significant educational decisions. Traditionally, the development of test items for such examinations has relied on manual development by subject matter experts. However, Automated Item Generation (AIG) using Large Language Models (LLMs) has emerged as a promising alternative, though systematic research on their application in high-stakes assessments, particularly in STEM fields like physics, is limited. High-stakes physics assessments must evaluate a range of cognitive skills, from basic recall to advanced analytical thinking. Previous AIG studies have predominantly focused on lower-order cognitive skills, neglecting higher-order thinking. Moreover, the psychometric quality of items generated by LLMs has not been thoroughly validated, raising concerns about their validity and reliability for high-stakes contexts. To explore this, we investigated LLM capabilities in generating physics test items suitable for Nigeria’s Unified Tertiary Matriculation Examination (UTME). Based on our preliminary findings, the current study utilized Gemini 2.0 flash and instructional prompting technique for item generation. Bloom’s taxonomy was used as a framework to generate items of different cognitive levels in line with the UTME blueprint. The quality of the generated items was evaluated through expert reviews using a six-criteria rubric and pilot testing with 527 final-year high school students. Psychometric analysis, based on Classical Test Theory and Item Response Theory, confirmed the high quality of the LLM-generated physics test, demonstrating unidimensionality and good psychometric properties. This study demonstrates that under controlled conditions and with expert review, LLMs can produce high-stakes physics MCQ items whose psychometric qualities (difficulty, discrimination) are comparable to those of a real UTME physics test. This suggests LLMs have potential as cost-effective item-generation tools, for educational institutions and testing organizations.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Can Large Language Models Develop High-Stakes Physics Exam Items? A Comprehensive Study of Cognitive and Psychometric Efficacy

  • 期刊论文

  • 发表日期：

    2026年1月8日

查看全文
  • Moses Oluoke Omopekunola,
  • Elena Yu. Kardanova

摘要

High-stakes assessment is crucial for evaluating student performance and making significant educational decisions. Traditionally, the development of test items for such examinations has relied on manual development by subject matter experts. However, Automated Item Generation (AIG) using Large Language Models (LLMs) has emerged as a promising alternative, though systematic research on their application in high-stakes assessments, particularly in STEM fields like physics, is limited. High-stakes physics assessments must evaluate a range of cognitive skills, from basic recall to advanced analytical thinking. Previous AIG studies have predominantly focused on lower-order cognitive skills, neglecting higher-order thinking. Moreover, the psychometric quality of items generated by LLMs has not been thoroughly validated, raising concerns about their validity and reliability for high-stakes contexts. To explore this, we investigated LLM capabilities in generating physics test items suitable for Nigeria’s Unified Tertiary Matriculation Examination (UTME). Based on our preliminary findings, the current study utilized Gemini 2.0 flash and instructional prompting technique for item generation. Bloom’s taxonomy was used as a framework to generate items of different cognitive levels in line with the UTME blueprint. The quality of the generated items was evaluated through expert reviews using a six-criteria rubric and pilot testing with 527 final-year high school students. Psychometric analysis, based on Classical Test Theory and Item Response Theory, confirmed the high quality of the LLM-generated physics test, demonstrating unidimensionality and good psychometric properties. This study demonstrates that under controlled conditions and with expert review, LLMs can produce high-stakes physics MCQ items whose psychometric qualities (difficulty, discrimination) are comparable to those of a real UTME physics test. This suggests LLMs have potential as cost-effective item-generation tools, for educational institutions and testing organizations.