Scholarlink

<p>Large Language Models (LLMs) are increasingly used in healthcare settings, yet concerns remain regarding their ability to safely and reliably handle medical dialogues. To address this issue, this study introduces a quantitative framework for evaluating the safety and trustworthiness of LLMs in multilingual medical dialogues. Using the German subtask of the NTCIR-18 MedNLP-CHAT dataset, we examined how 13 LLMs—encompassing general-purpose, open-source, and biomedical variants—identify medical, ethical, and legal risks. ROC–AUC–based statistical validation (one-sample t-tests and sign tests) were applied to ensure robust and reproducible evaluation. Results show that gpt-5, gpt-4o, gpt-3.5-Turbo, gpt-oss:120b, gpt-oss:20b, and gemma-3:27b consistently achieved reliable performance, while smaller and domain-specific models often failed to generalize across languages and risk types. These findings suggest that model scale and multi-domain safety alignment are key to achieving trustworthy risk reasoning in clinical dialogues and provide guidance for conservative deployment of LLMs in healthcare.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Towards Safe and Trustworthy Healthcare AI: Risk Assessment of Medical Dialogue Using LLMs

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年12月30日

查看全文
  • Hiroki Tanioka

摘要

Large Language Models (LLMs) are increasingly used in healthcare settings, yet concerns remain regarding their ability to safely and reliably handle medical dialogues. To address this issue, this study introduces a quantitative framework for evaluating the safety and trustworthiness of LLMs in multilingual medical dialogues. Using the German subtask of the NTCIR-18 MedNLP-CHAT dataset, we examined how 13 LLMs—encompassing general-purpose, open-source, and biomedical variants—identify medical, ethical, and legal risks. ROC–AUC–based statistical validation (one-sample t-tests and sign tests) were applied to ensure robust and reproducible evaluation. Results show that gpt-5, gpt-4o, gpt-3.5-Turbo, gpt-oss:120b, gpt-oss:20b, and gemma-3:27b consistently achieved reliable performance, while smaller and domain-specific models often failed to generalize across languages and risk types. These findings suggest that model scale and multi-domain safety alignment are key to achieving trustworthy risk reasoning in clinical dialogues and provide guidance for conservative deployment of LLMs in healthcare.