Scholarlink

<p>Accurate, context-appropriate likelihood ratios (LRs) are needed for Bayesian diagnosis, but empirical LRs are sparse because diagnostic accuracy studies are costly and context-dependent. We evaluated whether large language models (LLMs) can generate LR outputs that align with literature-reported values. We compared LR outputs generated by three OpenAI models (GPT-4o, o3, and GPT-5) with all literature-reported values curated in TheNNT.com dataset. A few-shot prompt was used to elicit numerical LR outputs, and agreement was assessed using Bland–Altman analyses to evaluate mean bias and multiplicative limits of agreement. A total of 700 reported LRs across 30 conditions were compiled, most involving signs or symptoms (59%), historical elements (19%), or test results (16%). All models demonstrated negligible mean bias. GPT-5 had the narrowest 95% limits of agreement (0.26×–3.70×) compared with o3 and GPT-4o. These findings indicate that LLM-generated LRs can be centered near literature values on average, but they are not interchangeable with published estimates for individual high-stakes decisions. Clinical use, including use for unstudied findings, would require prospective, context-specific diagnostic-accuracy validation.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Large language models generate diagnostic likelihood ratios with low mean bias but wide dispersion

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月11日

查看全文
  • Paul Chong,
  • Shuhan He,
  • Kian Samadian,
  • Amal Mohamed,
  • Boyu Peng,
  • Emma Chua,
  • Cory Rohlfsen,
  • Brian W. Locke

摘要

Accurate, context-appropriate likelihood ratios (LRs) are needed for Bayesian diagnosis, but empirical LRs are sparse because diagnostic accuracy studies are costly and context-dependent. We evaluated whether large language models (LLMs) can generate LR outputs that align with literature-reported values. We compared LR outputs generated by three OpenAI models (GPT-4o, o3, and GPT-5) with all literature-reported values curated in TheNNT.com dataset. A few-shot prompt was used to elicit numerical LR outputs, and agreement was assessed using Bland–Altman analyses to evaluate mean bias and multiplicative limits of agreement. A total of 700 reported LRs across 30 conditions were compiled, most involving signs or symptoms (59%), historical elements (19%), or test results (16%). All models demonstrated negligible mean bias. GPT-5 had the narrowest 95% limits of agreement (0.26×–3.70×) compared with o3 and GPT-4o. These findings indicate that LLM-generated LRs can be centered near literature values on average, but they are not interchangeable with published estimates for individual high-stakes decisions. Clinical use, including use for unstudied findings, would require prospective, context-specific diagnostic-accuracy validation.