Position Bias in LLMs for Critical Decision Support: A Case Study on Multiple Casualty Triage
摘要
Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as emergency response, medical triage, and security operations. This study uses a controlled, fictitious multiple casualty triage scenario to investigate the effects of position bias, the tendency of LLMs to prioritize information elements based on their position in a list rather than their relevance. For a list of patients, we vary the position of the most critically injured one to evaluate whether systematic deviation from the medically established START triage protocol can be observed for GPT-4o and GPT-4o mini. Our results reveal a consistent recency bias with both models: The most critically injured patient was less likely to be prioritized when listed first. This effect was more pronounced in shorter patient lists, challenging the common assumption that short prompts are less prone to evoke position bias. These findings raise critical concerns about the operational reliability of LLMs in time-sensitive, high-stakes tasks with high information density. Our study contributes to growing evidence that LLMs require rigorous validation before deployment in sensitive environments such as OSINT, defense informatics, and emergency triage.