Beam Search for Improvement of Code Generation in Answering Programming Questions with Code

We have developed a new approach to the beam search algorithm, which we have named parsing search. This approach is used in code generation with parsing of sequentially generated code. We tested the algorithm on programming problems from a question-answering community. Our implementation uses the CodeGen model as an underlying pre-trained language model (PLM), leading to improved consistency and quality of the generated code. To evaluate our approach, we used general-purpose and code-specific natural-language processing (NLP) metrics (Rouge, Ruby), as well as test-based evaluations (Run@1, Pass@1). Our evaluation was conducted on two publicly available datasets: HumanEval-X and APIzation, containing code snippets in Java, Python, JavaScript, and Go. The experimental results showed improvements compared to the direct application of PLM and the conventional beam search. We achieved significant improvement with Run@1 and Pass@1 (up to +173% compared to the direct “greedy” PLM application), demonstrating promising prospects for parsing search applications in various scenarios.