Validity of multiple-choice digital formative assessment for assessing students’ (mis)conceptions: evidence from a mixed-methods study in algebra

Assessing students’ (mis)conceptions is a challenging task for teachers as well as for researchers. While individual assessment, for example through interviews, can provide deep insights into students’ thinking, this is very time-consuming and therefore not feasible for whole classes or even larger settings. For those settings, automatically evaluated multiple-choice (MC) items could be a solution. However, it is a challenge to design those items and to adapt them for other countries in a way that they adequately reveal students’ (mis)conceptions. In this article, we investigate the question whether it is valid to use a German adaption of a multiple-choice test developed in Australia for formative assessment of the letter-as-object misconception in Germany. For this, first semi-structured interviews with five German Year 8 students were conducted, and second, 616 students were asked for short written explanations. These data were analysed with regards to the students’ (mis)conceptions and compared with their automatic online diagnosis. In general, a high concordance between online SMART test results and students’ explanations was observed, confirming that useful diagnoses of student misconceptions can be obtained from such a short well-designed MC test.