How Students Learn by Validating ChatGPT Responses

This study explores the hypothesis that students learn better when engaged in activities where they validate ChatGPT responses by contrasting them to reliable and valid human-generated content material. By applying an ecologically valid but not controlled experimental design we asked students to individually choose to work on an assignment either before (N = 80) or after (N = 42) a course written examination session. The assignment included four scenarios guiding students to interact with ChatGPT and evaluate afterwords the validity of the obtained responses. Available data indicate that students who worked on the assignment prior to examination were able to provide significantly improved answers to examination items that were conceptually relevant to the assignment tasks as compared to those irrelevant. This outcome, however, was valid only for open-ended questions included in the examination sheet and not for the closed-type ones. Overall, this study provides concrete research evidence that ChatGPT-like AI tools can provide the basis for designing beneficial learning activities, assuming the role of a “less competent partner” and offering to students the opportunity of critically reviewing their generated content. At theoretical level, we discuss how this perspective is in line with the proposed “AI as a black box” approach which bypasses the issues relevant to possible errors and misguidance generated by current level conversational AI technologies.