Scholarlink

Objectives <p>To compare the reliability, quality, and readability of artificial intelligence (AI)–generated patient information on vocal health and hygiene produced by three general-purpose large language models (LLMs): ChatGPT-5 (OpenAI), Grok-4 (xAI), and Claude System-4 (Anthropic).</p> Methods <p>Twenty-five frequently asked questions related to vocal health and hygiene were developed using established otolaryngology guidelines, major patient education resources, and clinical experience. Each question was submitted verbatim to all three LLMs under standardized prompting conditions. Responses were assessed for reliability (mDISCERN), information quality (EQIP), and readability (FRES, FKGL). Two otolaryngologists independently rated all outputs while blinded to the identity of the models. Inter-rater agreement was assessed using intraclass correlation coefficients (ICC)&#xa0;for EQIP and Cohen’s κ for mDISCERN.</p> Results <p>Claude demonstrated significantly higher mDISCERN and EQIP scores than ChatGPT and Grok (<i>p</i> &lt; 0.001), indicating higher reliability and information-quality scores within the evaluated FAQ set. Readability indices did not differ significantly among models (FRES, <i>p</i> = 0.510; FKGL, <i>p</i> = 0.590), and all outputs corresponded to the upper high school to college-level range (mean FKGL ≈ 12; low FRES values in the ‘difficult’ range), exceeding recommended health literacy thresholds. Subgroup analyses showed&#xa0;similar patterns across thematic domains. A strong positive correlation was observed between mDISCERN and EQIP (<i>ρ </i>= 0.72, <i>p</i> &lt; 0.001).</p> Conclusion <p>Under standardized prompting conditions, AI chatbots generated&#xa0;structured and internally consistent patient information text; however, the elevated reading level limits accessibility for the general public. Among the evaluated LLMs, Claude produced higher reliability and quality scores within this structured question set. These systems may serve as supplementary educational resources; however, professional clinical evaluation and individualized patient assessment remain essential.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Reliability, quality, and readability of AI-generated vocal health and hygiene information: a cross-sectional comparative study

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月9日

查看全文
  • Ebru Karakaya Gojayev,
  • İbrahim Türle,
  • Serhan Cömertoğlu

摘要

Objectives

To compare the reliability, quality, and readability of artificial intelligence (AI)–generated patient information on vocal health and hygiene produced by three general-purpose large language models (LLMs): ChatGPT-5 (OpenAI), Grok-4 (xAI), and Claude System-4 (Anthropic).

Methods

Twenty-five frequently asked questions related to vocal health and hygiene were developed using established otolaryngology guidelines, major patient education resources, and clinical experience. Each question was submitted verbatim to all three LLMs under standardized prompting conditions. Responses were assessed for reliability (mDISCERN), information quality (EQIP), and readability (FRES, FKGL). Two otolaryngologists independently rated all outputs while blinded to the identity of the models. Inter-rater agreement was assessed using intraclass correlation coefficients (ICC) for EQIP and Cohen’s κ for mDISCERN.

Results

Claude demonstrated significantly higher mDISCERN and EQIP scores than ChatGPT and Grok (p < 0.001), indicating higher reliability and information-quality scores within the evaluated FAQ set. Readability indices did not differ significantly among models (FRES, p = 0.510; FKGL, p = 0.590), and all outputs corresponded to the upper high school to college-level range (mean FKGL ≈ 12; low FRES values in the ‘difficult’ range), exceeding recommended health literacy thresholds. Subgroup analyses showed similar patterns across thematic domains. A strong positive correlation was observed between mDISCERN and EQIP (ρ = 0.72, p < 0.001).

Conclusion

Under standardized prompting conditions, AI chatbots generated structured and internally consistent patient information text; however, the elevated reading level limits accessibility for the general public. Among the evaluated LLMs, Claude produced higher reliability and quality scores within this structured question set. These systems may serve as supplementary educational resources; however, professional clinical evaluation and individualized patient assessment remain essential.