Scholarlink

<p>Large language models (LLMs) are increasingly applied in healthcare, yet their evaluation relies predominantly on static benchmarks that are costly, contamination-prone, and lack calibrated measurement properties for fine-grained performance tracking. We developed and validated a computerized adaptive testing (CAT) framework grounded in item response theory to enable scalable, psychometrically rigorous assessment of standardized medical knowledge in LLMs. A two-phase study comprising Monte Carlo simulations and empirical evaluation of 38 LLMs was conducted between July and September 2025. The CAT protocol achieved a near-perfect correlation with full-bank results (<i>r</i> = 0.988) using only 1.3% of items. Evaluation time decreased from 6.85 hours to 8.4 min per model, and token usage dropped from 1.77 million to 0.03 million. Model rankings were fully preserved (Spearman’s <i>ρ</i> = 1.0). At current API pricing, per-model evaluation costs fell from approximately $1,475 to under $5. This adaptive methodology serves as an essential pre- screening and continuous monitoring tool under a standardized testing protocol. Crucially, it is not a substitute for real-world clinical validation or safety-oriented prospective studies; rather, it enables developers and healthcare institutions to implement rigorous, high-frequency, evidence-based evaluation of foundational knowledge prior to more resource-intensive downstream testing.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月18日

查看全文
  • Tianpeng Zheng,
  • Zhehan Jiang,
  • Jiayi Liu,
  • Shicong Feng

摘要

Large language models (LLMs) are increasingly applied in healthcare, yet their evaluation relies predominantly on static benchmarks that are costly, contamination-prone, and lack calibrated measurement properties for fine-grained performance tracking. We developed and validated a computerized adaptive testing (CAT) framework grounded in item response theory to enable scalable, psychometrically rigorous assessment of standardized medical knowledge in LLMs. A two-phase study comprising Monte Carlo simulations and empirical evaluation of 38 LLMs was conducted between July and September 2025. The CAT protocol achieved a near-perfect correlation with full-bank results (r = 0.988) using only 1.3% of items. Evaluation time decreased from 6.85 hours to 8.4 min per model, and token usage dropped from 1.77 million to 0.03 million. Model rankings were fully preserved (Spearman’s ρ = 1.0). At current API pricing, per-model evaluation costs fell from approximately $1,475 to under $5. This adaptive methodology serves as an essential pre- screening and continuous monitoring tool under a standardized testing protocol. Crucially, it is not a substitute for real-world clinical validation or safety-oriented prospective studies; rather, it enables developers and healthcare institutions to implement rigorous, high-frequency, evidence-based evaluation of foundational knowledge prior to more resource-intensive downstream testing.