Fine-Tuning CodeLlama to Fix Bugs

Fixing bugs in the wild is a challenging task. Automating it would help software development immensely. Recently, Deep Learning-based approaches, including Large Language Models, have made great progress in this direction. Unfortunately, not many of these models have been open sourced, and proprietary models often outperform open-sourced ones. In this paper we investigate whether the recently open-sourced model, CodeLlama, can be improved through fine-tuning to be competitive on the bugfixing task. We investigate two kinds of tasks: singular bugfixing and whole program bugfixing. For fine-tuning, we use code from GitHub for the former and Project CodeNet published by IBM Research for the latter. The base and fine-tuned models are evaluated on the QuixBugs dataset for singular bugfixing and a collection of incorrect assignment submissions coded by students for whole program bugfixing. We find that our fine-tuned models outperform ChatGPT (GPT-3.5) on singular bugfixing. On whole program bugixing, they come close in the number of incorrect programs fixed while corrupting only half as many.