“Ok Pal, we have to code that now”: interaction patterns of programming beginners with a conversational chatbot

Chatbots based on large language models are becoming an important tool in modern software development, yet little is known about how programming beginners interact with this new technology to write code and acquire new knowledge. Thus, we are missing key ingredients to develop guidelines on how to adopt chatbots for becoming productive at programming.

With our research, we aim at identifying these ingredients. Specifically, we want to understand how programming beginners use conversational chatbots when writing source code.

To this end, we study programming beginners’ interaction with a chatbot in a CS2 course while they were solving programming assignments. Additionally, we evaluate the correctness of submitted solutions and compare them to solutions of beginners who did not use a conversational chatbot.

We analyzed 756 prompts of 129 conversations, most of them focusing on code generation. Interestingly, conversations that contain prompts asking for debugging or testing of code are linked with higher success rates, indicating that deeper engagement with code leads to higher quality code. Moreover, prompts without sufficient context often lead to unsatisfying results.

While not surprising, this underpins the importance that programming beginners need to know how to use chatbots, instead of merely using it as code generators without investing time in code quality. Moreover, companies should employ prompt guidelines, in which code quality prompts might be enforced after a code generation prompt has been stated.