Scholarlink

Background <p>Electronic medical records impose significant documentation burdens. While proprietary Large Language Models (LLMs) offer automation, they raise data sovereignty and privacy concerns. This study evaluated the feasibility of using locally deployable open-source LLMs (specifically DeepSeek-V3) versus proprietary models for dental EMR generation.</p> Methods <p>A dataset of 190 de-identified dental EMRs was converted into synthetic Mandarin doctor-patient dialogues, serving as a high-context linguistic stress test. Six LLMs—two open-source (DeepSeek-V3, LLaMA-3.1-405B) and four proprietary (GPT-4, GPT-4o, Claude-3.7-Sonnet, Grok-3)—were evaluated using the structured “CRISPE Framework” and a baseline “Basic Prompt”. Assessment utilized a dual-track framework: automated metrics (<i>n</i> = 190) and blinded expert clinical evaluation using a 7-Dimensional Index (7DI) on a stratified subset (<i>n</i> = 50).</p> Results <p>The CRISPE strategy significantly outperformed the basic prompt across all models (Mean 7DI: 28.86 ± 2.12 vs. 26.91 ± 2.36; <i>p</i>&lt;.001). Under optimized prompting, DeepSeek-V3 achieved a mean 7DI score of 28.76 ± 2.04, and we did not detect statistically significant differences versus GPT-4o (29.22 ± 2.25; <i>p</i>&gt;.05) in this expert-rated sample. One-way ANOVA similarly did not detect overall between-model differences (F = 0.54, <i>p</i>=.744); however, non-significance should not be interpreted as equivalence, and larger prospective evaluations are warranted.</p> Conclusion <p>Structured prompt engineering is critical for enhancing AI-generated documentation. Open-source models demonstrated clinical performance comparable to proprietary leaders when optimally prompted, though larger validation studies are needed. These preliminary findings suggest the feasibility of privacy-preserving local deployment strategies, offering a potential pathway to democratize AI support in dental institutions without compromising data sovereignty.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating open-source LLMs for dental EMR generation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月18日

查看全文
  • Hao Wang,
  • Wen Du,
  • Bo Yang,
  • Mengyu Liu,
  • Chunwei Xu,
  • Wei Zhang,
  • Chenfan Xu,
  • Leyang He,
  • Wenbo Zhang,
  • Yao Yu,
  • Jianan Lin,
  • Xin Peng

摘要

Background

Electronic medical records impose significant documentation burdens. While proprietary Large Language Models (LLMs) offer automation, they raise data sovereignty and privacy concerns. This study evaluated the feasibility of using locally deployable open-source LLMs (specifically DeepSeek-V3) versus proprietary models for dental EMR generation.

Methods

A dataset of 190 de-identified dental EMRs was converted into synthetic Mandarin doctor-patient dialogues, serving as a high-context linguistic stress test. Six LLMs—two open-source (DeepSeek-V3, LLaMA-3.1-405B) and four proprietary (GPT-4, GPT-4o, Claude-3.7-Sonnet, Grok-3)—were evaluated using the structured “CRISPE Framework” and a baseline “Basic Prompt”. Assessment utilized a dual-track framework: automated metrics (n = 190) and blinded expert clinical evaluation using a 7-Dimensional Index (7DI) on a stratified subset (n = 50).

Results

The CRISPE strategy significantly outperformed the basic prompt across all models (Mean 7DI: 28.86 ± 2.12 vs. 26.91 ± 2.36; p<.001). Under optimized prompting, DeepSeek-V3 achieved a mean 7DI score of 28.76 ± 2.04, and we did not detect statistically significant differences versus GPT-4o (29.22 ± 2.25; p>.05) in this expert-rated sample. One-way ANOVA similarly did not detect overall between-model differences (F = 0.54, p=.744); however, non-significance should not be interpreted as equivalence, and larger prospective evaluations are warranted.

Conclusion

Structured prompt engineering is critical for enhancing AI-generated documentation. Open-source models demonstrated clinical performance comparable to proprietary leaders when optimally prompted, though larger validation studies are needed. These preliminary findings suggest the feasibility of privacy-preserving local deployment strategies, offering a potential pathway to democratize AI support in dental institutions without compromising data sovereignty.