Role-based evaluation of artificial intelligence chatbot responses in orthodontic emergency scenarios: accuracy, readability, understandability, and patient-oriented communication

This study investigated role-based differences in the accuracy, readability, understandability, and internal consistency of chatbot responses to orthodontic emergencies and examined the clinical implications of patient-oriented communication.

Twenty-three standardized orthodontic emergency scenarios were presented to four chatbots— ChatGPT-4o, Claude 3 Opus, Microsoft Copilot, and Gemini 2.5—using patient and orthodontist roles. Response accuracy was evaluated by expert orthodontists and research assistants using a 3-point Likert scale, while readability, understandability, and internal consistency were assessed with the Atesman index, Sonmez formula, and Cronbach’s α. Patient-role responses were descriptively analyzed using predefined communication dimensions to contextualize quantitative findings.

Significant chatbot-specific role-based differences were observed, with Claude 3 Opus (p = 0.001) and Gemini 2.5 (p = 0.023) showing higher accuracy in the orthodontist role. In the patient role, ChatGPT-4o and Claude 3 Opus showed the highest rates of correct information, while Claude 3 Opus had the highest rate in the orthodontist role. Patient-role responses were significantly more understandable than orthodontist-role responses (p < 0.05). ChatGPT-4o (α = 0.862) and Gemini 2.5 (α = 0.815) showed high internal consistency. Qualitative analysis indicated that patient-oriented responses frequently adopted a reassuring tone and emphasized temporary self-care strategies, potentially influencing perceived urgency in orthodontic emergencies.

Chatbot performance varied according to user role. Patient-oriented responses were more understandable despite similar readability across models and could influence perceptions of urgency and professional responsibility, highlighting the need for cautious framing of chatbot-generated information for orthodontic emergency guidance.

Chatbots can provide preliminary information in orthodontic emergencies; however, due to limitations in accuracy and consistency, they should be used only as supportive tools and should not replace professional clinical judgment.