Evaluating the Effectiveness of Comparison Activities in a CTAT Tutor for Algorithmic Thinking

Prior work has shown that novice programmers find algorithmic thinking challenging. However, to date there is little research on how to help students learn this skill. We investigated if comparison of programming examples helps novice programmers improve skills related to generation and modification of algorithms. As a testbed for our research, we implemented two versions of a CTAT tutoring system: one that presented pairs of examples and asked students to compare them (comparison tutor), and a second that presented examples sequentially (sequential tutor). Findings from an experimental study (N = 57) showed that ignoring condition, students learned from the tutors. The comparison tutor significantly improved procedural knowledge but with the caveat that Bayesian statistics did not provide strong evidence for the alternative hypothesis. In contrast, there was no significant effect of tutor version on procedural flexibility or transfer questions and Bayesian statics provided substantial evidence for the null hypothesis.