Scholarlink

<p>Large Language Models (LLMs) hold potential as clinical decision support tools, particularly when integrated with domain-specific knowledge. In radiology, there is limited research on LLMs for assessing imaging appropriateness. This study evaluates a contextualized GPT-4-based LLM’s performance in assessing the appropriateness of musculoskeletal MRI scan requests with standard models and different versions of optimization. The LLMs’ performances was also compared against human clinicians with varying experience (two radiology residents, two subspecialist attendings, an orthopaedic surgeon). Using a retrieval-augmented generation framework, the LLM was provided with a domain-specific knowledge base from 33 American College of Radiology Appropriateness Criteria guidelines. A test dataset of 70 fictional case scenarios was created, including cases with insufficient clinical information. Quantitative analysis using the McNemar mid-P test revealed that the optimized LLM achieved 92.86% accuracy, significantly outperforming the baseline model (61.43%, <i>P</i> &lt; .001) and the standard GPT-4 model (51.29%, <i>P</i> &lt; .001). The optimized model also excelled in identifying cases with insufficient clinical information. In comparison to human clinicians, the optimized LLM performed better than all but one radiologist. This study demonstrates that with contextualization and optimization, GPT-4-based LLMs can improve performance in assessing imaging appropriateness and show promise as clinical decision support tools in radiology.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

ChatGPT performance in assessing musculoskeletal MRI scan appropriateness based on ACR appropriateness criteria

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年2月28日

查看全文
  • Jin Rong Tan,
  • Daniel Y. Z. Lim,
  • Quan Le,
  • Gita Y. Karande,
  • Lai Peng Chan,
  • Yeong Huei Ng,
  • Daniel S. W. Ting,
  • Sudharsan Madhavan,
  • Hiok Yang Chan,
  • Anh N. T. Tran,
  • Yusheng Keefe Lai

摘要

Large Language Models (LLMs) hold potential as clinical decision support tools, particularly when integrated with domain-specific knowledge. In radiology, there is limited research on LLMs for assessing imaging appropriateness. This study evaluates a contextualized GPT-4-based LLM’s performance in assessing the appropriateness of musculoskeletal MRI scan requests with standard models and different versions of optimization. The LLMs’ performances was also compared against human clinicians with varying experience (two radiology residents, two subspecialist attendings, an orthopaedic surgeon). Using a retrieval-augmented generation framework, the LLM was provided with a domain-specific knowledge base from 33 American College of Radiology Appropriateness Criteria guidelines. A test dataset of 70 fictional case scenarios was created, including cases with insufficient clinical information. Quantitative analysis using the McNemar mid-P test revealed that the optimized LLM achieved 92.86% accuracy, significantly outperforming the baseline model (61.43%, P < .001) and the standard GPT-4 model (51.29%, P < .001). The optimized model also excelled in identifying cases with insufficient clinical information. In comparison to human clinicians, the optimized LLM performed better than all but one radiologist. This study demonstrates that with contextualization and optimization, GPT-4-based LLMs can improve performance in assessing imaging appropriateness and show promise as clinical decision support tools in radiology.