Exploring the Impact of LLM-Generated Feedback: Evaluation from Professors and Students in Data Science Courses

Providing students with exercises and timely feedback is pivotal in education, particularly in data science. This paper aims to assess the quality of three different types of feedback, one generated through traditional artificial intelligence methods called “schematic feedback”, and the other two with different prompts passed to a Large Language Model (LLM), called “discursive feedback” and “combined feedback”, the latter also exploiting the “schematic feedback”. The three types of feedback are evaluated from both instructors’ and students’ perspectives, in terms of the feedback’s ability to provide helpful suggestions on how to solve an exercise correctly. Instructors considered the “combined feedback” the most helpful feedback, compared to the “schematic feedback” and “discursive feedback”, which were not considered substantially different. Students used the “schematic feedback” and “combined feedback” during their formative assessment activities: (i) they experienced the feedback better than expected, (ii) students of the current academic year (who used the “combined feedback” in addition to the “schematic feedback”) reported an improved experience compared to students of the previous academic year (who used only the “schematic feedback”). In conclusion, we observed an improved quality of the feedback using LLM (reported by both instructors and students), as long as the prompt includes specific context, in our case, achieved by the “combined feedback”. This study underscores the importance of leveraging LLMs in educational settings to provide tailored and helpful feedback, ultimately fostering a more effective learning environment.