Decomposing Topic Relevance: A Multi-agent LLM Approach for Automated Essay Scoring and Feedback
摘要
In Chinese language education, compositions reflect students’ language and reasoning skills. Topic relevance assesses cue comprehension and content focus, thereby guiding instructional strategies. Current state-of-the-art Large Language Models (LLMs), including architectures such as GPT-4, tend to perform poorly in assessing topic relevance, often producing erroneous judgements and generating feedback that lacks practical applicability. This study proposes an innovative zero sample Automatic Essay Scoring(AES) framework that innovatively deconstructs the core assessment dimension of ‘Topic Relevance’ and divides it into ‘Degree of Topic Deviation’ and ‘grade adaptation requirements’ as two key subcharacteristics. At the same time, this study proposes a three-step strategy that utilises the capabilities of LLMs to progressively achieve composition content comprehension, multivariate evaluation and feedback generation, thereby compensating for the shortcomings of traditional Automatic Essay Comment Generation(AECG) systems. Ultimately, depending on the score, different levels of comments and suggestions are given. By systematically eliciting the assessment capabilities of the LLM, our approach performs well across multiple automated assessment metrics and manual assessments, demonstrating its great potential for real-world applications.