Identifying Gaps in Students’ Explanations of Code Using LLMs

This study investigates methods based on Large Language Models (LLMs) to identify gaps or missing parts in learners’ self-explanations. This work is part of our broader effort to automate the evaluation of students’ freely generated responses, which in this work are learners’ self-explanations of code examples during code comprehension activities. We experimented with two methods and four distinct LLMs in two distinct settings. One method prompts LLMs to identify gaps in learners’ self-explanations, whereas the other method relies on LLMs performing a sentence-level semantic similarity task to identify gaps. We evaluated these methods in two settings: (i) simulated data generated using LLMs and (ii) actual student data. Results revealed the semantic similarity method significantly improves task performance over the zero-shot prompting for gap identification (the holistic method), i.e., over the standard method of prompting LLMs to directly address the gap identification task.