MADCAP: A Multi-agent Deliberative Framework for Robust Assessment of Open-Ended Questions
摘要
The assessment of Open-Ended Questions (OEQs), while crucial for evaluating higher-order thinking, is persistently hampered by subjectivity and scalability challenges. Although Large Language Models (LLMs) show considerable potential for automating this task, their application is often limited by inconsistent outputs, a singular evaluative perspective, and an inability to adapt to nuanced criteria. This paper introduces MADCAP (Multi-Agent Deliberation with Cluster-Aware Pairwise-compared criteria), a novel framework designed to overcome these limitations. MADCAP operationalizes the principles of collective human intelligence by structuring the evaluation process into three synergistic stages: (1) it employs unsupervised clustering to establish context-aware evaluation environments; (2) it dynamically induces and weights cluster-specific criteria via an LLM-driven, AHP-inspired method; and (3) it deploys a panel of specialized agents to score answers through a multi-round deliberative protocol that promotes consensus. Experiments conducted on three OEQ datasets showed that MADCAP significantly enhanced the adaptability and contextual relevance of the assessments, with an average improvement of 7% across multiple metrics compared to a strong baseline. The core contribution of this work lies in shifting the evaluation paradigm from single-model direct assessment to a structured, multi-agent deliberative system, thereby advancing the objectivity, reliability, and sophistication of automated OEQ assessment.