Predicting Successful Programming Submissions Based on Critical Logic Blocks

We propose an approach to infer the critical logic blocks contained within student submissions that can influence the passing or failing of individual exercises in introductory programming courses. Given a programming exercise, we extract critical logic blocks from the abstract syntax trees (ASTs) of its submissions and use a bag-of-words approach to train a decision tree classifier to predict the pass/failure of a submission given the logic blocks present in its AST. We apply this technique to two streams of an introductory Python programming course for high-school students, constructing decision trees for 66 programming exercises based on several thousand submissions for each exercise. We obtain classifiers with F1 scores of 92.4% for the beginners and 88.2% for the intermediate stream. These trained models are highly interpretable and can provide a visualisation of the key logic blocks that are critical in the pass/failure of a problem. We explain how the models can assist educators in understanding the common valid approaches or failed attempts carried out by students when tackling a specific problem, and they may also serve as a guide for suggesting hints to struggling students.