Scholarlink

<p>Large language model (LLM)-based chatbots have been utilized across various healthcare domains and have garnered substantial attention. This study aimed to evaluate and compare the performance of several LLM-based chatbots with that of medical students in responding to neurology questions. This cross-sectional study, conducted in December 2025 in Iran. ChatGPT-5, Gemini 3, Copilot 2025, Perplexity, and 20 medical students responded to a neurology questionnaire. A confusion matrix was utilized to analyze the data. In this regard, four metrics—sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV)—as well as overall accuracy were calculated. Moreover, correlations examined chatbot performance against question characteristics (word count, context, format, type, modality, language). The study revealed that overall performance metrics for the evaluated chatbots significantly outperformed those of medical students (<i>p</i> &lt; 0.001). Among the evaluated chatbots, Copilot exhibited superior performance (0.88), followed by ChatGPT-5 (0.86), in terms of accuracy. Meanwhile, quantitative question types were associated with a significant reduction in chatbot performance (r = 0.470, p = 0.001). The study findings presented valuable insights results particularly pertinent to neurology, where chatbots can serve as supplementary tools for practitioners, enhancing diagnostic accuracy and clinical decision-making while adhering to established ethical standards. However, further research is required to provide more precise insights, particularly with a larger sample size of human participants.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Comparison of the performance of ChatGPT-5, Gemini 3, Copilot, Perplexity, and medical students in answering neurology questions: a cross-sectional study

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月4日

查看全文
  • Mohsen Khosravi,
  • Maryam Yousefi-Roobiyat,
  • Zahra Asghari,
  • Motahareh Nakhaei,
  • Fatemeh Khosravi

摘要

Large language model (LLM)-based chatbots have been utilized across various healthcare domains and have garnered substantial attention. This study aimed to evaluate and compare the performance of several LLM-based chatbots with that of medical students in responding to neurology questions. This cross-sectional study, conducted in December 2025 in Iran. ChatGPT-5, Gemini 3, Copilot 2025, Perplexity, and 20 medical students responded to a neurology questionnaire. A confusion matrix was utilized to analyze the data. In this regard, four metrics—sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV)—as well as overall accuracy were calculated. Moreover, correlations examined chatbot performance against question characteristics (word count, context, format, type, modality, language). The study revealed that overall performance metrics for the evaluated chatbots significantly outperformed those of medical students (p < 0.001). Among the evaluated chatbots, Copilot exhibited superior performance (0.88), followed by ChatGPT-5 (0.86), in terms of accuracy. Meanwhile, quantitative question types were associated with a significant reduction in chatbot performance (r = 0.470, p = 0.001). The study findings presented valuable insights results particularly pertinent to neurology, where chatbots can serve as supplementary tools for practitioners, enhancing diagnostic accuracy and clinical decision-making while adhering to established ethical standards. However, further research is required to provide more precise insights, particularly with a larger sample size of human participants.