Scholarlink

<p>To evaluate the agreement of LLMs with the Preferred Practice Patterns<sup>®</sup> (PPP) guidelines developed by the American Academy of Ophthalmology (AAO). Open questions based on the AAO PPP were submitted to five LLMs: GPT-o1 and GPT-4o by OpenAI, Claude 3.5 Sonnet by Anthropic, Gemini 1.5 Pro by Google, and DeepSeek-R1-Lite-Preview. Questions were classified as “open” or “confirmatory with positive/negative ground-truth answer”. Three blinded investigators classified responses as “concordant”, “undetermined”, or “discordant” compared to the AAO PPP. Undetermined and discordant answers were analyzed to assess harming potential for patients. Responses referencing peer-reviewed articles were reported. In total, 147 questions were submitted to the LLMs. Concordant answers were 135 (91.8%) for GPT-o1, 133 (90.5%) for GPT-4o, 136 (92.5%) for Claude 3.5 Sonnet, 124 (84.4%) for Gemini 1.5 Pro, and 119 (81.0%) for DeepSeek-R1-Lite-Preview (<i>P</i> = 0.006). The highest number of harmful answers was reported for Gemini 1.5 Pro (<i>n</i> = 6, 4.1%), followed by DeepSeek-R1-Lite-Preview (<i>n</i> = 5, 3.4%). Gemini 1.5 Pro was the most transparent model (86 references, 58.5%). Other LLMs referenced papers in 9.5–15.6% of their responses. LLMs can provide discordant answers compared to ophthalmology guidelines, potentially harming patients by delaying diagnosis or recommending suboptimal treatments.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Large language models provide discordant information compared to ophthalmology guidelines

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年7月1日

查看全文
  • Andrea Taloni,
  • Antonia Carmen Sangregorio,
  • Giuseppe Alessio,
  • Maria Angela Romeo,
  • Giulia Coco,
  • Linda Marie Louise Busin,
  • Andrea Sollazzo,
  • Vincenzo Scorcia,
  • Giuseppe Giannaccare

摘要

To evaluate the agreement of LLMs with the Preferred Practice Patterns® (PPP) guidelines developed by the American Academy of Ophthalmology (AAO). Open questions based on the AAO PPP were submitted to five LLMs: GPT-o1 and GPT-4o by OpenAI, Claude 3.5 Sonnet by Anthropic, Gemini 1.5 Pro by Google, and DeepSeek-R1-Lite-Preview. Questions were classified as “open” or “confirmatory with positive/negative ground-truth answer”. Three blinded investigators classified responses as “concordant”, “undetermined”, or “discordant” compared to the AAO PPP. Undetermined and discordant answers were analyzed to assess harming potential for patients. Responses referencing peer-reviewed articles were reported. In total, 147 questions were submitted to the LLMs. Concordant answers were 135 (91.8%) for GPT-o1, 133 (90.5%) for GPT-4o, 136 (92.5%) for Claude 3.5 Sonnet, 124 (84.4%) for Gemini 1.5 Pro, and 119 (81.0%) for DeepSeek-R1-Lite-Preview (P = 0.006). The highest number of harmful answers was reported for Gemini 1.5 Pro (n = 6, 4.1%), followed by DeepSeek-R1-Lite-Preview (n = 5, 3.4%). Gemini 1.5 Pro was the most transparent model (86 references, 58.5%). Other LLMs referenced papers in 9.5–15.6% of their responses. LLMs can provide discordant answers compared to ophthalmology guidelines, potentially harming patients by delaying diagnosis or recommending suboptimal treatments.