Automated coding of content and pedagogical content knowledge of mathematics using a multi-agent large language model
摘要
Despite the importance of content-specific knowledge for teaching, the task of measuring such knowledge, particularly pedagogical content knowledge (PCK), remains challenging. Scholars have been more successful in capturing the distinct nature of PCK when it has been assessed through open-ended response items, and such responses have been significantly linked to the quality of mathematics instruction and student learning. However, the use of open-ended items entails substantial resource and time demands, as it requires the training and ongoing calibration of human coders to ensure reliable coding of responses. Although scholars have explored technological approaches to automating coding, traditional automated methods have not achieved sufficient reliability for coding complex constructs, such as PCK. Advances in large language models (LLMs) offer new possibilities; however, it remains unclear whether existing LLMs are adequate or whether a purposefully designed LLM framework is required. In this study, we explored the potential of a multi-agent LLM, GradeOpt, which we developed to code mathematics content knowledge (CK) and PCK reliably, in comparison with commonly used automated coding approaches, including two natural language processing (NLP) models (RoBERTa and SBERT) and a widely used LLM (GPT-4o). Using data collected from a national sample of 268 U.S. middle school mathematics teachers who responded to a set of CK and PCK items on ratios and proportional reasoning, we found that the multi-agent LLM, GradeOpt, achieved substantial agreement overall (