A Study of LLM Generated Line-by-Line Explanations in the Context of Conversational Program Comprehension Tutoring Systems

This research paper explores to what extent large language models (LLMs) can generate line-by-line explanations of code examples used in intro-to-programming courses such as CS1 (Computer Science) and CS2. While it is known that LLMs can generate code explanations, a systematic analysis of those explanations and their appropriateness for instructional and learning purposes is needed, which is the goal of this paper. Specifically, the paper explores how different types of prompts impact the nature and quality of line-by-line explanations relative to human expert explanations. We report a quantitative and qualitative analysis that compares AI-generated explanations with explanations produced by human experts. Furthermore, we investigate to what degree LLM can generate explanations for learners of various levels of mastery.