Scholarlink

Background <p>This study evaluates the use of large language models (LLMs) in generating Patient Education Materials (PEMs) for dental scenarios, focusing on their reliability, readability, understandability, and actionability. The study aimed to assess the performance of four LLMs—ChatGPT-4.0, Claude 3.5 Sonnet, Gemini 1.5 Flash, and Llama 3.1–405b—in generating PEMs for four common dental scenarios.</p> Methods <p>A comparative analysis was conducted where five independent dental professionals assessed the materials using the Patient Education Materials Assessment Tool (PEMAT) to evaluate understandability and actionability. Readability was measured with Flesch Reading Ease and Level scores, and inter-rater reliability was assessed using Fleiss’ Kappa.</p> Results <p>Llama 3.1–405b demonstrated the highest inter-rater reliability (Fleiss’ Kappa: 0.78–0.89). ChatGPT-4.0 excelled in understandability, surpassing the PEMAT threshold of 70% in three of the four scenarios. Claude 3.5 Sonnet performed well in understandability for two scenarios but did not consistently meet the 70% threshold for actionability. ChatGPT-4.0 generated the longest responses, while Claude 3.5 Sonnet produced the shortest.</p> Conclusions <p>ChatGPT-4.0 demonstrated superior understandability, while Llama 3.1–405b achieved the highest inter-rater reliability. The findings indicate that further refinement and human intervention is necessary for LLM-generated content to meet the standards of effective patient education.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Assessing the power of AI: a comparative evaluation of large language models in generating patient education materials in dentistry

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年6月18日

查看全文
  • Gowri Sivaramakrishnan,
  • Maryam Almuqahwi,
  • Sufyan Ansari,
  • Mohammed Lubbad,
  • Emad Alagamawy,
  • Kannan Sridharan

摘要

Background

This study evaluates the use of large language models (LLMs) in generating Patient Education Materials (PEMs) for dental scenarios, focusing on their reliability, readability, understandability, and actionability. The study aimed to assess the performance of four LLMs—ChatGPT-4.0, Claude 3.5 Sonnet, Gemini 1.5 Flash, and Llama 3.1–405b—in generating PEMs for four common dental scenarios.

Methods

A comparative analysis was conducted where five independent dental professionals assessed the materials using the Patient Education Materials Assessment Tool (PEMAT) to evaluate understandability and actionability. Readability was measured with Flesch Reading Ease and Level scores, and inter-rater reliability was assessed using Fleiss’ Kappa.

Results

Llama 3.1–405b demonstrated the highest inter-rater reliability (Fleiss’ Kappa: 0.78–0.89). ChatGPT-4.0 excelled in understandability, surpassing the PEMAT threshold of 70% in three of the four scenarios. Claude 3.5 Sonnet performed well in understandability for two scenarios but did not consistently meet the 70% threshold for actionability. ChatGPT-4.0 generated the longest responses, while Claude 3.5 Sonnet produced the shortest.

Conclusions

ChatGPT-4.0 demonstrated superior understandability, while Llama 3.1–405b achieved the highest inter-rater reliability. The findings indicate that further refinement and human intervention is necessary for LLM-generated content to meet the standards of effective patient education.