Scholarlink

Background <p>Large language models (LLMs) are increasingly applied in medicine; however, their accuracy in guideline-driven, high-stakes specialties, such as metabolic and bariatric surgery (MBS), remains uncertain. This study evaluates the performance of ChatGPT-4o, Gemini 2.0 Flash, and DeepSeek-V3 in generating guideline-concordant responses to MBS clinical questions.</p> Methods <p>Thirty standardized, guideline-based MBS questions were presented to each model. Responses were randomized in order, anonymized (blinded as Model A/B/C), and evaluated by 93 MBS experts using a validated 0–3 scale (0 = inaccurate; 3 = fully guideline-concordant). A repeated-measures ANOVA with Bonferroni correction tested model differences; reliability was assessed with Cronbach’s α and intraclass correlation coefficients (ICC).</p> Results <p>DeepSeek-V3 achieved the highest mean score (2.44 ± 0.40), followed by ChatGPT-4o (1.79 ± 0.46) and Gemini 2.0 Flash (1.63 ± 0.47) (<i>p</i> &lt; 0.001). Fully guideline-concordant ratings (score = 3) were most frequent for DeepSeek (80%) vs. ChatGPT (0%) and Gemini (3.3%). Internal consistency was excellent (α &gt; 0.90), and inter-rater reliability was strong (ICC &gt; 0.88). When mapped against the QUEST evaluation framework, the study addressed Quality and Understanding but did not fully capture Expression, Safety, or Trust dimensions.</p> Conclusions <p>DeepSeek-V3 outperformed ChatGPT-4o and Gemini 2.0 Flash in generating guideline-concordant responses in MBS. These results highlight the need for ongoing, domain-focused validation before clinical use.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Accuracy and Knowledge Base Evaluation of ChatGPT-4o, Gemini-2.0-Flash, and DeepSeek-V3 in Metabolic and Bariatric Surgery: an Expert-Rated Blinded Study

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月25日

查看全文
  • Mohamed Hany,
  • Mohamed H. Zidan,
  • Chetan Parmar,
  • Shahab Shahabi Shahmiri,
  • Hashem Altabbaa,
  • Ahmed El-Shamarka,
  • Ahmed Amgad,
  • Islam M. Abdelkhalek,
  • Abdullah A. Assal,
  • Marwan Emad Abdou,
  • Mohammad Kermansaravi,
  • Abdelrahman Nimeri,
  • Adel Abou-mrad,
  • Ahmed Abokhozima,
  • Ala Wafa,
  • Amir Davarpanah Jazi,
  • André Lázaro,
  • Andrea Schroeder,
  • Andrew G Robertson,
  • Angelo Iossa,
  • Anıl Ergin,
  • Anna Casajoana,
  • Anwar Ashraf Abouelnasr,
  • Aparna Govil Bhasker,
  • Ashraf Haddad,
  • Asim Shabbir,
  • Benjamin Clapp,
  • Carlos Augusto Scussel Madalosso,
  • Carlos Padrón,
  • Cem Emir Guldogan,
  • Christine Stier,
  • Cüneyt Kirkil,
  • Daniel Moritz Felsenreich,
  • Richa Jaiswal,
  • Ebrahim Aghajani,
  • Estuardo Behrens,
  • Farah A. Husain,
  • Farnaz Rahimi,
  • Ghulam Siddiq,
  • Giovanni Lezoche,
  • Heykel Mebarek,
  • Hosam Mohamed Mostafa Elghadban,
  • Ivaylo Tzvetkov,
  • Karl Peter Rheinwalt,
  • Kazunori Kasama,
  • Levon N. Grigoryan,
  • Maria Antonieta Barrera,
  • Mariano Palermo,
  • Masoud Rezvani,
  • Massimiliano Di Paola,
  • Michael Talbot,
  • Michel Gagner,
  • Michel Vix,
  • Miguel-A Carbajo,
  • Mohamad Hayssam Elfawal,
  • Mohamed Ibrahim Bahnasy,
  • Mohamed Mokhtar Arafat,
  • Mohamed Ammar,
  • Mousa Khoursheed,
  • Natan Zundel,
  • Nikolaos Pararas,
  • Nuru Bayramov,
  • Otto Montoya,
  • Panagiotis Lainas,
  • Paolo Gentileschi,
  • Patrick Noel,
  • Paulina Salminen,
  • Piotr Major,
  • Ramen Goel,
  • Rob Snoekx,
  • Rodolfo J. Rodolfo,
  • Rodrigue Chemaly,
  • Rudolf Weiner,
  • Rui José Silva Ribeiro,
  • Ruth Blackham,
  • Salvatore Tolone,
  • Samer G. Mattar,
  • Sara Gaafar Ibnauf Suliman,
  • Sergio Carandina,
  • Sergio O. Aparicio,
  • Sergio Verboonen,
  • Silvana Leanza,
  • Silvia Leite,
  • Sjaak Pouwels,
  • Sonja Chiappetta,
  • Stefano Olmi,
  • Suhaib Ahmad,
  • Tadeja Pintar,
  • Tarek Hassab,
  • Tigran Poghosyan,
  • Tuna Bilecik,
  • Valdemir José Alegre Salles,
  • Vasileios Charalampakis,
  • Wah Yang,
  • Yannick Nijs,
  • Yves Borbély

摘要

Background

Large language models (LLMs) are increasingly applied in medicine; however, their accuracy in guideline-driven, high-stakes specialties, such as metabolic and bariatric surgery (MBS), remains uncertain. This study evaluates the performance of ChatGPT-4o, Gemini 2.0 Flash, and DeepSeek-V3 in generating guideline-concordant responses to MBS clinical questions.

Methods

Thirty standardized, guideline-based MBS questions were presented to each model. Responses were randomized in order, anonymized (blinded as Model A/B/C), and evaluated by 93 MBS experts using a validated 0–3 scale (0 = inaccurate; 3 = fully guideline-concordant). A repeated-measures ANOVA with Bonferroni correction tested model differences; reliability was assessed with Cronbach’s α and intraclass correlation coefficients (ICC).

Results

DeepSeek-V3 achieved the highest mean score (2.44 ± 0.40), followed by ChatGPT-4o (1.79 ± 0.46) and Gemini 2.0 Flash (1.63 ± 0.47) (p < 0.001). Fully guideline-concordant ratings (score = 3) were most frequent for DeepSeek (80%) vs. ChatGPT (0%) and Gemini (3.3%). Internal consistency was excellent (α > 0.90), and inter-rater reliability was strong (ICC > 0.88). When mapped against the QUEST evaluation framework, the study addressed Quality and Understanding but did not fully capture Expression, Safety, or Trust dimensions.

Conclusions

DeepSeek-V3 outperformed ChatGPT-4o and Gemini 2.0 Flash in generating guideline-concordant responses in MBS. These results highlight the need for ongoing, domain-focused validation before clinical use.