Scholarlink

<p>The emergence of Large Language Models (LLMs) within the Traditional Chinese Medicine (TCM) domain presents an urgent need to assess their clinical application capabilities. However, such evaluations are challenged by the individualized, holistic, and diverse nature of TCM’s “Syndrome Differentiation and Treatment” (SDT). Existing benchmarks are confined to knowledge-based question-answering or the accuracy of syndrome differentiation, often neglecting assessment of treatment decision-making. Here, we propose a comprehensive benchmark derived from clinical cases, classical case records, and authoritative examination questions. Designated TCM-BEST4SDT, the dataset comprises 600 questions covering four tasks: TCM Basic Knowledge, Medical Ethics, LLM Content Safety, and SDT. Data annotation adheres to a rigorous three-stage pipeline consisting of expert annotation, mutual cross-validation, and independent third-party review. The evaluation framework integrates three mechanisms, namely selected-response evaluation, judge model evaluation, and a specialized reward model employed to quantify prescription-syndrome congruence. Crucially, a controlled evaluation on the Qwen3 series demonstrated a strictly monotonic correlation between performance and model scale, verifying the benchmark’s sensitivity in discriminating model capabilities. This study establishes a standardized evaluation framework for intelligent TCM research, objectifies the personalized diagnostic logic of complementary medicine, and facilitates the optimization and application of large language models in digital medicine.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年8月3日

查看全文
  • Kunning Li,
  • Jianbin Guo,
  • Zhaoyang Shang,
  • Yiqing Liu,
  • Hongmin Du,
  • Lingling Liu,
  • Yuping Zhao,
  • Lifeng Dong

摘要

The emergence of Large Language Models (LLMs) within the Traditional Chinese Medicine (TCM) domain presents an urgent need to assess their clinical application capabilities. However, such evaluations are challenged by the individualized, holistic, and diverse nature of TCM’s “Syndrome Differentiation and Treatment” (SDT). Existing benchmarks are confined to knowledge-based question-answering or the accuracy of syndrome differentiation, often neglecting assessment of treatment decision-making. Here, we propose a comprehensive benchmark derived from clinical cases, classical case records, and authoritative examination questions. Designated TCM-BEST4SDT, the dataset comprises 600 questions covering four tasks: TCM Basic Knowledge, Medical Ethics, LLM Content Safety, and SDT. Data annotation adheres to a rigorous three-stage pipeline consisting of expert annotation, mutual cross-validation, and independent third-party review. The evaluation framework integrates three mechanisms, namely selected-response evaluation, judge model evaluation, and a specialized reward model employed to quantify prescription-syndrome congruence. Crucially, a controlled evaluation on the Qwen3 series demonstrated a strictly monotonic correlation between performance and model scale, verifying the benchmark’s sensitivity in discriminating model capabilities. This study establishes a standardized evaluation framework for intelligent TCM research, objectifies the personalized diagnostic logic of complementary medicine, and facilitates the optimization and application of large language models in digital medicine.