Scholarlink

<p>Artificial intelligence has been widely used to answer questions in the medical context. This study aimed to evaluate the performance, reliability, and precision of ChatGPT-4.0 in responding to multiple-choice questions (MCQs) previously administered to medical students. We conducted an observational and cross-sectional study to assess the performance of ChatGPT by analyzing its accuracy, examining associations with specific knowledge areas and Bloom’s taxonomy levels, assessing the influence of the psychometric properties of the items, and investigating the effect of images on the results. From the eight examinations analyzed, chatbot performance varied from 46.7 to 90.0% on the first attempt, 47.5 to 90% on the second attempt, and 28.3 to 89.2% on the third attempt. The concordance rate varied from 56.2% to 62.0% with Cohen’s kappa coefficients ranging from 0.071 to 0.217. On the second and third attempts, basic science had the highest scores (90.0 and 93.3%, respectively), whereas surgery (55.8%) and pediatrics (43.4%) had the lowest scores. In summary, the chatbot demonstrated poor performance that was inferior to its human counterparts in medical examinations and low reliability and precision in answering medical questions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Factors Associated with Lower Performance of Artificial Intelligence on Answering Undergraduate Medical Education Multiple-Choice Questions

  • 期刊论文

  • 发表日期：

    2025年5月26日

查看全文
  • Renato Ferretti,
  • Joyce Santana Rizzi,
  • Lorraine Silva Requena,
  • Angelica Maria Bicudo,
  • Pedro Tadao Hamamoto Filho

摘要

Artificial intelligence has been widely used to answer questions in the medical context. This study aimed to evaluate the performance, reliability, and precision of ChatGPT-4.0 in responding to multiple-choice questions (MCQs) previously administered to medical students. We conducted an observational and cross-sectional study to assess the performance of ChatGPT by analyzing its accuracy, examining associations with specific knowledge areas and Bloom’s taxonomy levels, assessing the influence of the psychometric properties of the items, and investigating the effect of images on the results. From the eight examinations analyzed, chatbot performance varied from 46.7 to 90.0% on the first attempt, 47.5 to 90% on the second attempt, and 28.3 to 89.2% on the third attempt. The concordance rate varied from 56.2% to 62.0% with Cohen’s kappa coefficients ranging from 0.071 to 0.217. On the second and third attempts, basic science had the highest scores (90.0 and 93.3%, respectively), whereas surgery (55.8%) and pediatrics (43.4%) had the lowest scores. In summary, the chatbot demonstrated poor performance that was inferior to its human counterparts in medical examinations and low reliability and precision in answering medical questions.