Scholarlink

Objectives <p>This study aimed to comprehensively evaluate the readability, quality, and reliability of the content generated by four state-of-the-art artificial intelligence (AI)-powered chatbots—OpenAI ChatGPT-4, DeepSeek v3, Google Gemini 2.5 Pro, and Grok-2—in response to frequently asked questions (FAQs) posed by patients on the Internet regarding chronic otitis media (COM). </p> Methods <p>A curated set of 25 frequently asked questions (FAQs) on COM was meticulously compiled using insights from Google Trends, Semrush, and authoritative clinical sources. Each question was posed to four leading AI-powered chatbots—OpenAI ChatGPT-4, DeepSeek v3, Google Gemini 2.5 Pro, and Grok-2—via standardized prompts to ensure consistency. Responses were rigorously evaluated for quality and reliability using the Ensuring Quality Information for Patients (EQIP) tool and the modified DISCERN (mDISCERN) instrument. Readability was assessed through the Flesch Reading Ease Score (FRES) and the Flesch–Kincaid Grade Level (FKGL), offering a comprehensive appraisal of the clarity, accuracy, and patient-centeredness of AI-generated content.</p> Results <p>Statistically significant differences were observed among the evaluated chatbots in terms of information quality and reliability, as reflected by EQIP and mDISCERN scores (<i>p</i> &lt; 0.001), with Grok demonstrating higher performance. In contrast, no statistically significant differences were found in readability measures (FRES and FKGL), and all responses required advanced reading skills. Moreover, no statistically significant differences were observed among the evaluated AI models in terms of readability.</p> Conclusion <p>AI-powered chatbots show promise in delivering health-related information about chronic otitis media; however, their outputs vary notably in quality and reliability while remaining uniformly challenging in terms of readability. Ongoing validation and optimization are essential to enhance the accessibility and educational value of AI-generated medical content.</p> Level of evidence <p>5.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

How smart are the machines? An analysis of AI responses on chronic otitis media

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月15日

查看全文
  • Abdullah Musa Altas,
  • Serhan Comertoglu,
  • Ibrahim Turle,
  • Ebru Karakaya Gojayev

摘要

Objectives

This study aimed to comprehensively evaluate the readability, quality, and reliability of the content generated by four state-of-the-art artificial intelligence (AI)-powered chatbots—OpenAI ChatGPT-4, DeepSeek v3, Google Gemini 2.5 Pro, and Grok-2—in response to frequently asked questions (FAQs) posed by patients on the Internet regarding chronic otitis media (COM).

Methods

A curated set of 25 frequently asked questions (FAQs) on COM was meticulously compiled using insights from Google Trends, Semrush, and authoritative clinical sources. Each question was posed to four leading AI-powered chatbots—OpenAI ChatGPT-4, DeepSeek v3, Google Gemini 2.5 Pro, and Grok-2—via standardized prompts to ensure consistency. Responses were rigorously evaluated for quality and reliability using the Ensuring Quality Information for Patients (EQIP) tool and the modified DISCERN (mDISCERN) instrument. Readability was assessed through the Flesch Reading Ease Score (FRES) and the Flesch–Kincaid Grade Level (FKGL), offering a comprehensive appraisal of the clarity, accuracy, and patient-centeredness of AI-generated content.

Results

Statistically significant differences were observed among the evaluated chatbots in terms of information quality and reliability, as reflected by EQIP and mDISCERN scores (p < 0.001), with Grok demonstrating higher performance. In contrast, no statistically significant differences were found in readability measures (FRES and FKGL), and all responses required advanced reading skills. Moreover, no statistically significant differences were observed among the evaluated AI models in terms of readability.

Conclusion

AI-powered chatbots show promise in delivering health-related information about chronic otitis media; however, their outputs vary notably in quality and reliability while remaining uniformly challenging in terms of readability. Ongoing validation and optimization are essential to enhance the accessibility and educational value of AI-generated medical content.

Level of evidence

5.