MIMIC-RxBench: Benchmarking Large Language Models for Prescription Error Classification
摘要
Accurate prescribing is vital for patient safety, yet errors can cause severe harm. Large language model (LLM)-based clinical decision support systems (CDSS) show promise in detecting such errors, but their performance in complex settings like ICU polypharmacy remains underexplored. Current evaluation datasets are often oversimplified, with limited error types and poor public availability. Moreover, class imbalance and rarity of key medication errors complicate reliable model evaluation. To address these challenges, we propose a hybrid LLM-human error injection framework and construct MIMIC-RxBench, the first benchmark for prescription error classification based on real ICU discharge records. It covers seven error types plus a “No Error” category, with strict controls on plausibility, context complexity, and sample diversity, substantially enhancing both difficulty and granularity. We benchmark various LLMs and prompt strategies, finding that while performance improves with model size, the best model reaches only a 0.462 micro-F1 score, underscoring the challenge posed by this benchmark. Models with stronger reasoning or medical specialization perform better, and prompt effectiveness varies across model types.