Scholarlink

Background <p>This study evaluates the comparative performance of DeepSeek-R1, ChatGPT-5 Pro, and urologists in the specific domains of bladder cancer knowledge retrieval and complex clinical reasoning.</p> Methods <p>We constructed a benchmark dataset comprising 91 standardized multiple-choice questions on bladder cancer derived from MedQA, MedMCQA, and the Chinese National Medical Licensing Examination, alongside three retrospectively reconstructed real-world cases. Five advanced models, including DeepSeek (V3, R1) and OpenAI variants (ChatGPT-5, ChatGPT-5 Pro, ChatGPT-5 Mini), were evaluated. Accuracy and stability were assessed across three independent runs for standardized questions. In clinical simulations, DeepSeek-R1 and ChatGPT-5 Pro were benchmarked against human urologists. A blinded expert panel of three senior urologists evaluated responses using a 5-point Likert scale across four dimensions: readability, medical accuracy, diagnostic test appropriateness, and logical coherence.</p> Results <p>In standardized testing, all models achieved &gt;92% accuracy. ChatGPT-5 Pro ranked first (97.52%), followed closely by DeepSeek-R1 (95.04%), with both displaying superior stability. In clinical simulations, DeepSeek-R1 demonstrated logical coherence comparable to both human experts and ChatGPT-5 Pro (<i>P</i> &gt; 0.05). However, DeepSeek-R1 scored significantly lower than ChatGPT-5 Pro regarding readability and "error-free" rates. Notably, human urologists significantly outperformed both AI models in diagnostic test appropriateness (<i>P</i> &lt; 0.01), primarily because DeepSeek-R1 struggled with "test avoidance," tending to recommend redundant investigations (<i>P</i> &lt; 0.0001).</p> Conclusions <p>DeepSeek-R1 demonstrates excellent accuracy and expert-level clinical reasoning, exhibiting competitiveness with ChatGPT-5 Pro. Although slightly inferior in readability and prone to suggesting unnecessary tests, its core reasoning capabilities remain favorable.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Benchmarking Large Language Models for Bladder Cancer: Comparative Knowledge Retrieval and Clinical Reasoning of DeepSeek and ChatGPT

  • 期刊论文

  • 发表日期：

    2026年6月14日

查看全文
  • Lei Peng,
  • Yi Zhang,
  • Anguo Zhao,
  • Rongkang Li,
  • Zhilin Li,
  • Jiadong Zhao,
  • Peng Yu,
  • Yang Liu,
  • Hongjin Shi,
  • Rui Liang,
  • Haifeng Wang

摘要

Background

This study evaluates the comparative performance of DeepSeek-R1, ChatGPT-5 Pro, and urologists in the specific domains of bladder cancer knowledge retrieval and complex clinical reasoning.

Methods

We constructed a benchmark dataset comprising 91 standardized multiple-choice questions on bladder cancer derived from MedQA, MedMCQA, and the Chinese National Medical Licensing Examination, alongside three retrospectively reconstructed real-world cases. Five advanced models, including DeepSeek (V3, R1) and OpenAI variants (ChatGPT-5, ChatGPT-5 Pro, ChatGPT-5 Mini), were evaluated. Accuracy and stability were assessed across three independent runs for standardized questions. In clinical simulations, DeepSeek-R1 and ChatGPT-5 Pro were benchmarked against human urologists. A blinded expert panel of three senior urologists evaluated responses using a 5-point Likert scale across four dimensions: readability, medical accuracy, diagnostic test appropriateness, and logical coherence.

Results

In standardized testing, all models achieved >92% accuracy. ChatGPT-5 Pro ranked first (97.52%), followed closely by DeepSeek-R1 (95.04%), with both displaying superior stability. In clinical simulations, DeepSeek-R1 demonstrated logical coherence comparable to both human experts and ChatGPT-5 Pro (P > 0.05). However, DeepSeek-R1 scored significantly lower than ChatGPT-5 Pro regarding readability and "error-free" rates. Notably, human urologists significantly outperformed both AI models in diagnostic test appropriateness (P < 0.01), primarily because DeepSeek-R1 struggled with "test avoidance," tending to recommend redundant investigations (P < 0.0001).

Conclusions

DeepSeek-R1 demonstrates excellent accuracy and expert-level clinical reasoning, exhibiting competitiveness with ChatGPT-5 Pro. Although slightly inferior in readability and prone to suggesting unnecessary tests, its core reasoning capabilities remain favorable.