Scholarlink

Background <p>Patients increasingly turn to the Internet and, more recently, to Large Language Models (LLMs) to seek answers to dental questions. However, the safety of LLM-generated dental advice for patients has not been systematically assessed.</p> Methods <p>Thirty dental questions spanning six disciplines were developed and posed in Polish and English to six LLMs. Each question was answered five times per model per language, yielding 2,700 unique LLM-generated answers, each independently evaluated by two dentists. A binary text classifier combining Qwen3-Embedding-0.6B word embeddings with a Support Vector Machine (SVM) was trained on 1,680 labeled answers and evaluated on 420 held-out answers.</p> Results <p>Substantial inter-rater agreement on harmfulness was observed (Cohen’s <InlineEquation ID="IEq1"><EquationSource Format="TEX">\(\kappa = 0.722\)</EquationSource></InlineEquation>). Without a system prompt, 56% of outputs from <Emphasis FontCategory="NonProportional">gpt-4o</Emphasis>, <Emphasis FontCategory="NonProportional">gpt-4o-mini</Emphasis>, and <Emphasis FontCategory="NonProportional">llama-3.3-70b</Emphasis> were classified as harmful (pooling both languages; a question was counted as harmful if at least one of its five generated answers was labeled harmful by at least one annotator). A safety-oriented system prompt reduced this rate by 26 percentage points to 30%. <Emphasis FontCategory="NonProportional">llama-3.3-70b</Emphasis> showed markedly higher harmfulness in Polish than English (<InlineEquation ID="IEq2"><EquationSource Format="TEX">\(\chi ^2 = 25.65\)</EquationSource></InlineEquation>, <InlineEquation ID="IEq3"><EquationSource Format="TEX">\(p &lt; 0.000001\)</EquationSource></InlineEquation>). The trained classifier showed a tendency to mark non-harmful answers as harmful, though performance was lower for Polish than English outputs.</p> Conclusions <p>Publicly available LLMs can pose potential safety risks to dental patients when used without guardrails. System prompts significantly mitigate harmful outputs. Language-specific model training and hallucination reduction strategies, such as retrieval-augmented generation, are recommended as future directions.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A cross-sectional evaluation of Large Language Model answers to dental questions

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月11日

查看全文
  • Martyna Mysior,
  • Marek Piotr Mysior,
  • Pamela Maslowski,
  • Katarzyna Skośkiewicz-Malinowska

摘要

Background

Patients increasingly turn to the Internet and, more recently, to Large Language Models (LLMs) to seek answers to dental questions. However, the safety of LLM-generated dental advice for patients has not been systematically assessed.

Methods

Thirty dental questions spanning six disciplines were developed and posed in Polish and English to six LLMs. Each question was answered five times per model per language, yielding 2,700 unique LLM-generated answers, each independently evaluated by two dentists. A binary text classifier combining Qwen3-Embedding-0.6B word embeddings with a Support Vector Machine (SVM) was trained on 1,680 labeled answers and evaluated on 420 held-out answers.

Results

Substantial inter-rater agreement on harmfulness was observed (Cohen’s \(\kappa = 0.722\)). Without a system prompt, 56% of outputs from gpt-4o, gpt-4o-mini, and llama-3.3-70b were classified as harmful (pooling both languages; a question was counted as harmful if at least one of its five generated answers was labeled harmful by at least one annotator). A safety-oriented system prompt reduced this rate by 26 percentage points to 30%. llama-3.3-70b showed markedly higher harmfulness in Polish than English (\(\chi ^2 = 25.65\), \(p < 0.000001\)). The trained classifier showed a tendency to mark non-harmful answers as harmful, though performance was lower for Polish than English outputs.

Conclusions

Publicly available LLMs can pose potential safety risks to dental patients when used without guardrails. System prompts significantly mitigate harmful outputs. Language-specific model training and hallucination reduction strategies, such as retrieval-augmented generation, are recommended as future directions.