A Code Analysis Tool to Help Students in the Age of Generative AI

In light of the fast-paced progress of Generative AI tools, we noticed our students’ interest in using generated code for their course assignments. We evaluate how well someone can solve computer programming problems with ChatGPT-4 using problems that were previously given in our exams at the University. We also analyse how well such solutions perform compared to real-life solutions produced by our cohort of students before the release of ChatGPT-4. The capability of our code evaluation tool to assess code correctness and code quality proves to be insightful to expose limitations of generated code, and we outline several possible improvements to increase this capability.