Evaluation Metrics in LLM Code Generation

The advanced capabilities of large language models can also be seen in their increasing use in the automatic generation of programming code. Although models are generally getting better and better, there are very few metrics that can be used to evaluate the quality of the generated code. In particular, this evaluation becomes challenging without dependence on good reference data in the form of tests or alternative solutions. In this paper, we explore both existing and new approaches to evaluate generated python code. These approaches can be classified into two categories: similarity-based and reference independent. The similarity-based approaches involve examining the code’s syntax-tree structure and embeddings and comparing them to reference code from the dataset. On the other hand, the reference independent approaches utilize static code analysis metrics used to assess human-written code. These metrics include maintainability and adherence to style guidelines. We examine these metrics on the example of several state-of-the-art code generation models to test their validity. Based on our results, the independent metrics seem to be the most promising approaches for future research.