Generative AI in Data Science Programming: Differences in Performance Between Groups With and Without AI-assistance
摘要
There has been a surge in interest in AI assistance as part of various processes. This trend has attracted scholarly attention from multiple disciplines. Notably, research on AI-assistance in data science programming remains nascent, with limited understanding of how AI-assistance influences learner outcomes. The purpose of this study was to ascertain if AI-assistance mattered for data science programming performance. We compared differences in performance on three machine learning classification problems between two groups: one group received AI assistance and another did not. Our results indicate that AI-assistance had an impact for only one of the three problems. Among the three dependent variables (performance on the three problems), there was a statistically significant difference in adjusted means for performance on the second problem, but not for the first or third problem. For the second problem, performance was statistically significantly greater in the No-AI group compared to the AI group. The findings suggest that the impact of AI-assistance on learning outcomes is influenced by various factors. The implications of the study are that AI assistance has the potential to be beneficial when used effectively. Our findings also underscore the importance of understanding the underlying mechanisms by which AI influences learner outcomes. The study and its findings open new perspectives to better understand data science learning and performance in the context of the interplay of contemporary AI applications with learners.