Scholarlink

<p>Large language models (LLMs) may support patients by addressing their queries; however, their real-world clinical use remains uncertain. Patients’ queries prospectively collected and answered by nuclear medicine physicians, administrative staffs, and ChatGPT v4.1. Responses evaluated and Likert-scored by medical and administrative experts and two independent non-experts using 15 out of 17 dimensions of the proposed QUEST framework. LLM-generated responses classified as “better,” “equivalent,” or “worse” relative to human-generated responses on question-by-question basis; binomial tests assessed whether LLM performance exceeded 50%. Inter-rater agreement was assessed using Prevalence-Adjusted Bias-Adjusted Kappa (PABAK); statistical significance set at <i>p</i> &lt; 0.05. 339 drug interaction, 42 medical, and 76 administrative queries were analysed. For medical queries, in 8 of 10 dimensions, 76–98% of LLM-generated responses rated equivalent or better than human-generated responses by medical expert (<i>p</i> &lt; 0.001). For administrative queries, non-expert raters judged LLM-generated responses more informative (97%) and preferred (86%). For medical queries, LLM-generated responses rated more informative (67%), human-generated responses judged easier to understand (62%), with 60% disagreement on overall preference. PABAK showed higher agreement for LLM- than human-generated responses across medical (0.14–0.90 vs −0.90–0.52) and administrative (0.92–1.00 vs −0.63– − 0.13) queries. LLM-generated responses were consistently rated favourably, particularly for administrative queries, though further validation is required before clinical use.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Real-world evaluation of large language model for patients medical and administrative queries in nuclear medicine

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月17日

查看全文
  • Arash Latifoltojar,
  • Akintunde Orunmuyi,
  • Michael Wang,
  • Stefan Adrian Vöö,
  • Irfan Kayani,
  • Richard Oyibo,
  • Kabita Gurung,
  • Jennifer Williams,
  • Memuna Rashid,
  • Andre Lopes,
  • Ashley Macallistar Groves,
  • Jamshed Bomanji

摘要

Large language models (LLMs) may support patients by addressing their queries; however, their real-world clinical use remains uncertain. Patients’ queries prospectively collected and answered by nuclear medicine physicians, administrative staffs, and ChatGPT v4.1. Responses evaluated and Likert-scored by medical and administrative experts and two independent non-experts using 15 out of 17 dimensions of the proposed QUEST framework. LLM-generated responses classified as “better,” “equivalent,” or “worse” relative to human-generated responses on question-by-question basis; binomial tests assessed whether LLM performance exceeded 50%. Inter-rater agreement was assessed using Prevalence-Adjusted Bias-Adjusted Kappa (PABAK); statistical significance set at p < 0.05. 339 drug interaction, 42 medical, and 76 administrative queries were analysed. For medical queries, in 8 of 10 dimensions, 76–98% of LLM-generated responses rated equivalent or better than human-generated responses by medical expert (p < 0.001). For administrative queries, non-expert raters judged LLM-generated responses more informative (97%) and preferred (86%). For medical queries, LLM-generated responses rated more informative (67%), human-generated responses judged easier to understand (62%), with 60% disagreement on overall preference. PABAK showed higher agreement for LLM- than human-generated responses across medical (0.14–0.90 vs −0.90–0.52) and administrative (0.92–1.00 vs −0.63– − 0.13) queries. LLM-generated responses were consistently rated favourably, particularly for administrative queries, though further validation is required before clinical use.