Scholarlink

<p>Code translation between programming languages (PLs) is a long-existing and critical task in software engineering, facilitating the modernization of legacy systems, ensuring cross-platform compatibility, and enhancing software performance. With the recent advances in Large Language Models (LLMs) and their applications to code translation, there is an increasing need for comprehensive evaluation of these models. Most existing studies instruct LLMs to perform code translation and evaluate their performance by either running the generated outputs through test suites or comparing them to reference outputs (ground truth). These outputs, however, may contain not only executable source code but also additional non-code elements, such as natural language explanations or formatting tokens. We refer to the way source code and non-code elements are combined as <i>output format</i>. It is crucial to understand and address variations in output format, as non-code elements can interfere with evaluation metrics, resulting in biased (inaccurate or unfair) assessments of model performance and comparisons. We refer to this bias as <i>output format bias</i>. To investigate the presence of output format biases, we first conduct an empirical analysis of the outputs from eleven instruct-tuned open-source LLMs, applied to 3,820 translation pairs across five languages: C, C++, Go, Java, and Python. The results show that between 26.4% and 73.7% of outputs produced by our evaluated LLMs necessitate post-processing (to result in source code for evaluation). To mitigate output format bias, we propose a strategic combination of prompt engineering and regular expressions that effectively extracts source code from mixed-format outputs, enabling the eleven open-source models to achieve an average Code Extraction Success Rate (CSR) of 92.73%. Our empirical study confirms that output format bias affects widely used execution-based metrics, i.e., Computational Accuracy (CA), and text-based metrics, i.e., BLEU, CodeBLEU and CrystalBLEU. Additionally, we test five closed LLMs and observe that they also generate varying distributions of output formats, which could contribute to output format biases. Our results highlight the need to mitigate the output format bias to enable reliable evaluations in LLMs code translation.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Output format biases in the evaluation of large language models for code translation

  • 期刊论文

  • 发表日期：

    2025年12月6日

查看全文
  • Marcos Macedo,
  • Yuan Tian,
  • Filipe R. Cogo,
  • Bram Adams

摘要

Code translation between programming languages (PLs) is a long-existing and critical task in software engineering, facilitating the modernization of legacy systems, ensuring cross-platform compatibility, and enhancing software performance. With the recent advances in Large Language Models (LLMs) and their applications to code translation, there is an increasing need for comprehensive evaluation of these models. Most existing studies instruct LLMs to perform code translation and evaluate their performance by either running the generated outputs through test suites or comparing them to reference outputs (ground truth). These outputs, however, may contain not only executable source code but also additional non-code elements, such as natural language explanations or formatting tokens. We refer to the way source code and non-code elements are combined as output format. It is crucial to understand and address variations in output format, as non-code elements can interfere with evaluation metrics, resulting in biased (inaccurate or unfair) assessments of model performance and comparisons. We refer to this bias as output format bias. To investigate the presence of output format biases, we first conduct an empirical analysis of the outputs from eleven instruct-tuned open-source LLMs, applied to 3,820 translation pairs across five languages: C, C++, Go, Java, and Python. The results show that between 26.4% and 73.7% of outputs produced by our evaluated LLMs necessitate post-processing (to result in source code for evaluation). To mitigate output format bias, we propose a strategic combination of prompt engineering and regular expressions that effectively extracts source code from mixed-format outputs, enabling the eleven open-source models to achieve an average Code Extraction Success Rate (CSR) of 92.73%. Our empirical study confirms that output format bias affects widely used execution-based metrics, i.e., Computational Accuracy (CA), and text-based metrics, i.e., BLEU, CodeBLEU and CrystalBLEU. Additionally, we test five closed LLMs and observe that they also generate varying distributions of output formats, which could contribute to output format biases. Our results highlight the need to mitigate the output format bias to enable reliable evaluations in LLMs code translation.