On the Helpfulness of a Zero-Shot Socratic Tutor
摘要
An AI tutor with a backend of GPT-4 was used as a homework assistant in an introductory data science course with 127 students. The tutor was given the homework assignments and the solutions, but was instructed not to give answers directly, and instead reply to student queries with questions meant to lead the student in the right direction, deploying a strategy known as the Socratic method. All interactions were anonymously logged and coded on seventeen attributes such as helpfulness and degree of answer leak. Surveys were also deployed to better understand students’ perceptions and use cases. The tutor was generally perceived as helpful to learning by students and an independent coder, and only rarely directly leaked the solution. However, the system also displayed a pattern of student achievement observed in other tutoring systems, where students who asked too many questions did worse on the midterms, even controlling for programming background. Other trends in the usage of the tutor are also discussed.