Scholarlink

Problem. LLMs are being piloted for clinical coding and decision support, yet no open benchmark targets the hospital-funding layer where Diagnosis-Related Groups (DRGs) determine reimbursement. In most OECD systems, DRGs route a substantial share of multi-trillion-dollar health spending through governed grouper software, making transparency and auditability first-order concerns rather than mere implementation details. Research Question. To what extent can LLMs emulate, explain, and apply DRG-based hospital payment rules? Contribution. NordDRG-AI-Benchmark is released, the first public, rule-complete test-bed for DRG reasoning. It bundles (i) machine-readable \(\sim \!20\) -sheet NordDRG definition tables and (ii) expert manuals and change-log templates that capture governance workflows, and exposes two suites: a 13-task Logic benchmark (code lookup, cross-table inference, grouping features, multilingual terminology, and CC/MCC validity checks) and a 13-task Grouper benchmark that requires full DRG-grouper emulation with strict exact-match scoring on both the DRG and the triggering drg_logic.id. Lightweight reference agents (LogicAgent, GrouperAgent) enable artefact-only evaluation. Results. Under an artefact-only (no-web) setting, on the 13 Logic tasks GPT-5 Thinking and Opus 4.1 score 13/13, o3 12/13; mid-tier models (GPT-5 Thinking Mini, o4-mini, GPT-5 Fast) achieve 6–8/13, and the remaining models score 5/13 or below. On full grouper emulation across 13 tasks (exact match on both DRG and drg_logic.id), GPT-5 Thinking solves 7/13, o3 6/13, and o4-mini 3/13; GPT-5 Thinking Mini solves 1/13, and all other tested endpoints score 0/13. Answer to RQ: Top-tier LLMs reliably master rule-level DRG reasoning; the strongest models can partially emulate the full DRG grouping logic. To available knowledge, this is the first public report of an LLM partially emulating the complete NordDRG grouper logic with governance-grade traceability. Significance. Coupling a rule-complete release with exact-match tasks and open scoring reveals domain-specific strengths and weaknesses that generic leaderboards miss and provides a reproducible yardstick for head-to-head and longitudinal evaluation in hospital funding. Availability. All artefacts and scripts are available at https://github.com/longshoreforrest/norddrg-ai-benchmark .

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

NordDRG AI Benchmark for Large Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Tapio Pitkäranta

摘要

Problem. LLMs are being piloted for clinical coding and decision support, yet no open benchmark targets the hospital-funding layer where Diagnosis-Related Groups (DRGs) determine reimbursement. In most OECD systems, DRGs route a substantial share of multi-trillion-dollar health spending through governed grouper software, making transparency and auditability first-order concerns rather than mere implementation details. Research Question. To what extent can LLMs emulate, explain, and apply DRG-based hospital payment rules? Contribution. NordDRG-AI-Benchmark is released, the first public, rule-complete test-bed for DRG reasoning. It bundles (i) machine-readable \(\sim \!20\) -sheet NordDRG definition tables and (ii) expert manuals and change-log templates that capture governance workflows, and exposes two suites: a 13-task Logic benchmark (code lookup, cross-table inference, grouping features, multilingual terminology, and CC/MCC validity checks) and a 13-task Grouper benchmark that requires full DRG-grouper emulation with strict exact-match scoring on both the DRG and the triggering drg_logic.id. Lightweight reference agents (LogicAgent, GrouperAgent) enable artefact-only evaluation. Results. Under an artefact-only (no-web) setting, on the 13 Logic tasks GPT-5 Thinking and Opus 4.1 score 13/13, o3 12/13; mid-tier models (GPT-5 Thinking Mini, o4-mini, GPT-5 Fast) achieve 6–8/13, and the remaining models score 5/13 or below. On full grouper emulation across 13 tasks (exact match on both DRG and drg_logic.id), GPT-5 Thinking solves 7/13, o3 6/13, and o4-mini 3/13; GPT-5 Thinking Mini solves 1/13, and all other tested endpoints score 0/13. Answer to RQ: Top-tier LLMs reliably master rule-level DRG reasoning; the strongest models can partially emulate the full DRG grouping logic. To available knowledge, this is the first public report of an LLM partially emulating the complete NordDRG grouper logic with governance-grade traceability. Significance. Coupling a rule-complete release with exact-match tasks and open scoring reveals domain-specific strengths and weaknesses that generic leaderboards miss and provides a reproducible yardstick for head-to-head and longitudinal evaluation in hospital funding. Availability. All artefacts and scripts are available at https://github.com/longshoreforrest/norddrg-ai-benchmark .