Scholarlink

Background <p>Carpal tunnel syndrome (CTS) is the most common peripheral nerve compression disorder and a frequent indication for neurosurgical consultation. Patients increasingly obtain medical information from artificial intelligence (AI) chatbots and social media platforms prior to clinical evaluation. The comparative quality and safety of these digital information sources in CTS remain insufficiently defined. The aim of this study was to systematically compare the quality of AI-generated medical advice with social media influencer–generated content regarding CTS, using a blinded, multi-rater evaluation framework. Specific objectives were to (i) quantify differences in accuracy, safety, clarity, and evidence-based alignment between the two information sources; (ii) characterize platform-level variability among social media sources; and (iii) evaluate inter-rater reliability of expert assessment.</p> Methods <p>In this blinded comparative study, twenty-five patient-oriented CTS questions were developed to reflect common inquiries encountered in neurosurgical and peripheral nerve practice. Matched responses were generated from ChatGPT (GPT-4) and from high-reach social media influencers across TikTok, Instagram, and YouTube (total responses <i>n</i> = 50). Three independent expert raters evaluated each response across four domains—accuracy, safety, clarity, and evidence-based alignment—using a 0–5 scale per domain (total score 0–20). Paired statistical analyses were performed, and inter-rater reliability was assessed using intraclass correlation coefficients (ICC).</p> Results <p>AI-generated responses demonstrated significantly higher total quality scores than social media influencer content (17.1 ± 0.9 vs. 12.4 ± 1.4; <i>p</i> &lt; 0.001; Cohen’s d = 3.23). Superior performance was observed across all domains: accuracy (4.3 ± 0.3 vs. 3.1 ± 0.5), safety (4.4 ± 0.4 vs. 3.1 ± 0.6), clarity (4.2 ± 0.4 vs. 3.6 ± 0.5), and evidence-based alignment (4.1 ± 0.5 vs. 2.7 ± 0.8) (all <i>p</i> &lt; 0.001). Among platforms, YouTube demonstrated higher scores than Instagram and TikTok, although all remained inferior to AI-generated responses. Inter-rater reliability was good (overall ICC = 0.76).</p> Conclusions <p>In this blinded expert evaluation, AI-generated responses provided more accurate, safer, and more evidence-aligned CTS information than social media influencer content. While AI systems cannot replace clinical assessment, they may serve as structured adjuncts for patient education in peripheral nerve disorders. The variability observed in social media content underscores the need for neurosurgical engagement in guiding patients toward reliable digital health resources.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Blinded comparative evaluation of AI-generated and social media–based medical advice in carpal tunnel syndrome

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月15日

查看全文
  • Mohammed Safwat Hamza,
  • Amr Khafagy

摘要

Background

Carpal tunnel syndrome (CTS) is the most common peripheral nerve compression disorder and a frequent indication for neurosurgical consultation. Patients increasingly obtain medical information from artificial intelligence (AI) chatbots and social media platforms prior to clinical evaluation. The comparative quality and safety of these digital information sources in CTS remain insufficiently defined. The aim of this study was to systematically compare the quality of AI-generated medical advice with social media influencer–generated content regarding CTS, using a blinded, multi-rater evaluation framework. Specific objectives were to (i) quantify differences in accuracy, safety, clarity, and evidence-based alignment between the two information sources; (ii) characterize platform-level variability among social media sources; and (iii) evaluate inter-rater reliability of expert assessment.

Methods

In this blinded comparative study, twenty-five patient-oriented CTS questions were developed to reflect common inquiries encountered in neurosurgical and peripheral nerve practice. Matched responses were generated from ChatGPT (GPT-4) and from high-reach social media influencers across TikTok, Instagram, and YouTube (total responses n = 50). Three independent expert raters evaluated each response across four domains—accuracy, safety, clarity, and evidence-based alignment—using a 0–5 scale per domain (total score 0–20). Paired statistical analyses were performed, and inter-rater reliability was assessed using intraclass correlation coefficients (ICC).

Results

AI-generated responses demonstrated significantly higher total quality scores than social media influencer content (17.1 ± 0.9 vs. 12.4 ± 1.4; p < 0.001; Cohen’s d = 3.23). Superior performance was observed across all domains: accuracy (4.3 ± 0.3 vs. 3.1 ± 0.5), safety (4.4 ± 0.4 vs. 3.1 ± 0.6), clarity (4.2 ± 0.4 vs. 3.6 ± 0.5), and evidence-based alignment (4.1 ± 0.5 vs. 2.7 ± 0.8) (all p < 0.001). Among platforms, YouTube demonstrated higher scores than Instagram and TikTok, although all remained inferior to AI-generated responses. Inter-rater reliability was good (overall ICC = 0.76).

Conclusions

In this blinded expert evaluation, AI-generated responses provided more accurate, safer, and more evidence-aligned CTS information than social media influencer content. While AI systems cannot replace clinical assessment, they may serve as structured adjuncts for patient education in peripheral nerve disorders. The variability observed in social media content underscores the need for neurosurgical engagement in guiding patients toward reliable digital health resources.