Language-Model Assisted Learning How to Program?

Foundational language models have forever changed how NLP prototypes may be rapidly constructed, dramatically reducing the “cost of curiosity”. This also affects the way we can teach and learn how to program. In this paper, we explore how well foundational models such as large, pre-trained neural transformers can answer questions pertaining to programming in a “learning to code” context: we present a new dataset comprising questions that students that learn how to program and students of particular programming languages – as offered by typical undergraduate university courses – typically ask. We cover both fundamental concepts in programming and also specific programming language issues. Although our study focuses on English, we believe results would be similar for other human languages due to the multilingual nature of many foundational language models. We explore how well a foundational (generic) pre-trained language model can answer them. To the best of our knowledge, this is one of the first studies that assesses how well generic foundational models and applications like ChatGPT are capable of answering different types of typical programming-related questions. This is a question of primary importance if we consider using such models to assist human students in their struggle to become (good) programmers.