Scholarlink

<p>Large Language Models (LLMs) have made significant advancements in code-related tasks. In the field of automated debugging, fault localization (FL) and automated program repair (APR) are two prevalent topics attracting significant research effort. Recently, in the field of FL and APR, many novel LLM-based approaches have emerged. However, most existing LLM-based studies primarily focus on the GPT models from OpenAI or open-source LLMs. With the rapid development of LLMs, various internet giants have introduced new closed-source models. In addition, due to policy restrictions, some regions can only access the commercial LLMs provided by specified companies. Despite the LLMs of OpenAI, the effectiveness of the other closed-source LLMs in FL and APR remains unknown. To better understand the effectiveness of contemporary closed-source models, we conduct a large-scale empirical study on their performance with respect to FL and APR. Specifically, our study involves 4 recent commercial closed-source LLMs (i.e., GPT-4o-Mini, Ernie-3.5, Qwen-turbo, and Doubao-pro) and 1 open-source LLM (i.e., DeepSeek-V3-chat). Note that only the GPT models have region restrictions among all LLMs we studied. We designed a total of 12 distinct prompt templates, 6 each for FL and APR, incorporating various formats and information sources. We conducted experiments to evaluate the effectiveness of FL and APR on 1036 real Java bugs from two datasets, Defects4J 2.0 and ConDefects. The key findings of the experiments indicate that (1) different LLMs tend to succeed on different sets of bugs in both FL and APR, with relatively little overlap among successful cases, implying the models possess distinct strengths in handling specific kinds of bugs, (2) the effectiveness of prompt templates varies across different models, and (3) the effectiveness of FL and APR capabilities of the studied models is significantly correlated with the bug type. We summarized all 14 findings obtained into 3 implications, which could help researchers further improve the performance of LLMs on FL and APR.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Assessing the effectiveness of recent closed-source large language models in fault localization and automated program repair

  • 期刊论文

  • 发表日期：

    2025年10月28日

查看全文
  • Bo Wang,
  • Ming Deng,
  • Mingda Chen,
  • Youfang Lin,
  • Jianyi Zhou,
  • Jie M. Zhang

摘要

Large Language Models (LLMs) have made significant advancements in code-related tasks. In the field of automated debugging, fault localization (FL) and automated program repair (APR) are two prevalent topics attracting significant research effort. Recently, in the field of FL and APR, many novel LLM-based approaches have emerged. However, most existing LLM-based studies primarily focus on the GPT models from OpenAI or open-source LLMs. With the rapid development of LLMs, various internet giants have introduced new closed-source models. In addition, due to policy restrictions, some regions can only access the commercial LLMs provided by specified companies. Despite the LLMs of OpenAI, the effectiveness of the other closed-source LLMs in FL and APR remains unknown. To better understand the effectiveness of contemporary closed-source models, we conduct a large-scale empirical study on their performance with respect to FL and APR. Specifically, our study involves 4 recent commercial closed-source LLMs (i.e., GPT-4o-Mini, Ernie-3.5, Qwen-turbo, and Doubao-pro) and 1 open-source LLM (i.e., DeepSeek-V3-chat). Note that only the GPT models have region restrictions among all LLMs we studied. We designed a total of 12 distinct prompt templates, 6 each for FL and APR, incorporating various formats and information sources. We conducted experiments to evaluate the effectiveness of FL and APR on 1036 real Java bugs from two datasets, Defects4J 2.0 and ConDefects. The key findings of the experiments indicate that (1) different LLMs tend to succeed on different sets of bugs in both FL and APR, with relatively little overlap among successful cases, implying the models possess distinct strengths in handling specific kinds of bugs, (2) the effectiveness of prompt templates varies across different models, and (3) the effectiveness of FL and APR capabilities of the studied models is significantly correlated with the bug type. We summarized all 14 findings obtained into 3 implications, which could help researchers further improve the performance of LLMs on FL and APR.