Background <p>The integration of artificial intelligence (AI) chatbots in medicine is expanding rapidly, with notable models like ChatGPT by OpenAI, Gemini by Google, and Copilot by Microsoft. These chatbots are increasingly used to provide medical information, yet their reliability in specific areas such as cardiovascular imaging remains underexplored. This study aims to evaluate the accuracy and reliability of ChatGPT (versions 3.5 and 4), Gemini, and Copilot in responding to patient inquiries about cardiovascular imaging.</p> Methods <p>We sourced 30 patient-oriented questions on cardiovascular imaging. The questions were submitted to ChatGPT-4, ChatGPT-3.5, Copilot Balanced Mode, Copilot Precise Mode, and Gemini. Responses were evaluated by two cardiovascular radiologists based on accuracy, clarity, completeness, neutrality, and appropriateness using a structured rubric. Inter-rater reliability was assessed using Cohen’s Kappa.</p> Results <p>ChatGPT-4 achieved the highest performance with 78.3% accuracy, 86.87% clarity and appropriateness, 81.7% completeness, and 100% neutrality. Gemini showed balanced performance, while Copilot Balanced Mode excelled in clarity and accuracy but lagged in completeness. Copilot Precise Mode had the lowest scores in completeness and accuracy. Penalty assessments revealed that ChatGPT-4 had the lowest incidence of missing or misleading information.</p> Conclusion <p>ChatGPT-4 emerged as the most reliable AI model for providing accurate, clear, and comprehensive patient information on cardiovascular imaging. While other models showed potential, they require further refinement. This study underscores the value of integrating AI chatbots into clinical practice to enhance patient education and engagement.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating the accuracy and reliability of AI chatbots in patient education on cardiovascular imaging: a comparative study of ChatGPT, gemini, and copilot

  • Ahmed Marey,
  • Abdelrahman M. Saad,
  • Yousef Tanas,
  • Hossam Ghorab,
  • Julia Niemierko,
  • Hazif Backer,
  • Muhammad Umair

摘要

Background

The integration of artificial intelligence (AI) chatbots in medicine is expanding rapidly, with notable models like ChatGPT by OpenAI, Gemini by Google, and Copilot by Microsoft. These chatbots are increasingly used to provide medical information, yet their reliability in specific areas such as cardiovascular imaging remains underexplored. This study aims to evaluate the accuracy and reliability of ChatGPT (versions 3.5 and 4), Gemini, and Copilot in responding to patient inquiries about cardiovascular imaging.

Methods

We sourced 30 patient-oriented questions on cardiovascular imaging. The questions were submitted to ChatGPT-4, ChatGPT-3.5, Copilot Balanced Mode, Copilot Precise Mode, and Gemini. Responses were evaluated by two cardiovascular radiologists based on accuracy, clarity, completeness, neutrality, and appropriateness using a structured rubric. Inter-rater reliability was assessed using Cohen’s Kappa.

Results

ChatGPT-4 achieved the highest performance with 78.3% accuracy, 86.87% clarity and appropriateness, 81.7% completeness, and 100% neutrality. Gemini showed balanced performance, while Copilot Balanced Mode excelled in clarity and accuracy but lagged in completeness. Copilot Precise Mode had the lowest scores in completeness and accuracy. Penalty assessments revealed that ChatGPT-4 had the lowest incidence of missing or misleading information.

Conclusion

ChatGPT-4 emerged as the most reliable AI model for providing accurate, clear, and comprehensive patient information on cardiovascular imaging. While other models showed potential, they require further refinement. This study underscores the value of integrating AI chatbots into clinical practice to enhance patient education and engagement.