Scholarlink

<p>In recent years, large language models (LLMs) have emerged<!--Query ID="Q1" Text="Please check if the author group, affiliations, corresponding authors email, authors email and corresponding authors are captured and presented correctly." Resolved="yes"--> and reshaped software engineering tasks such as code completion. Although LLMs have shown great performance on code completion, their performance on low-resource languages like Cangjie shows significant limitations. The main challenge is the limited availability of training data for new programming languages, resulting in under-fitting on the syntax and semantic information of the code. In this paper, we introduce <i>CodeBridge</i>, an innovative approach designed for code completion tasks of low-resource programming languages like Cangjie. <i>CodeBridge</i> proposes a dual approach to leveraging cross-language knowledge: a three-stage continued pretraining strategy to explicitly transfer knowledge from similar high-resource languages, and a novel chain-of-thought data generation method that leverages LLMs’ understanding of programming patterns combined with compiler feedback. Furthermore, <i>CodeBridge</i> proposes a prefix matching decoding strategy, which optimizes tokenization during inference to ensure consistency between training and inference. Our chain-of-thought approach addresses the fundamental trade-off between the high costs of generating compilable code and effective pretraining of code models. Our experiments demonstrate that <i>CodeBridge</i> achieves improvements in code completion for Cangjie at both line- and block-levels. Especially for DeepSeek-Coder-V2-Lite, our approach achieves 52.35% exact match at the line level and 33.27% line accuracy at the block level, representing improvements of 5.82% and 2.57%, respectively. In addition, the human-like chain-of-thought data provides an additional 1.01% improvement at the block level. We open-source the entire training pipeline, including data collection, cleaning, three-stage training, and model inference, offering a validated framework to support code tasks on future new programming languages.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Effective Fine-tuning for Low-resource Languages: A Case Study of Cangjie

  • 期刊论文

  • 发表日期：

    2026年5月23日

查看全文
  • Zhihao Lin,
  • Zhaofeng Liu,
  • Mingyi Zhou,
  • Zihan Huang,
  • Chi Chen,
  • Wei Ma,
  • Li Li

摘要

In recent years, large language models (LLMs) have emerged and reshaped software engineering tasks such as code completion. Although LLMs have shown great performance on code completion, their performance on low-resource languages like Cangjie shows significant limitations. The main challenge is the limited availability of training data for new programming languages, resulting in under-fitting on the syntax and semantic information of the code. In this paper, we introduce CodeBridge, an innovative approach designed for code completion tasks of low-resource programming languages like Cangjie. CodeBridge proposes a dual approach to leveraging cross-language knowledge: a three-stage continued pretraining strategy to explicitly transfer knowledge from similar high-resource languages, and a novel chain-of-thought data generation method that leverages LLMs’ understanding of programming patterns combined with compiler feedback. Furthermore, CodeBridge proposes a prefix matching decoding strategy, which optimizes tokenization during inference to ensure consistency between training and inference. Our chain-of-thought approach addresses the fundamental trade-off between the high costs of generating compilable code and effective pretraining of code models. Our experiments demonstrate that CodeBridge achieves improvements in code completion for Cangjie at both line- and block-levels. Especially for DeepSeek-Coder-V2-Lite, our approach achieves 52.35% exact match at the line level and 33.27% line accuracy at the block level, representing improvements of 5.82% and 2.57%, respectively. In addition, the human-like chain-of-thought data provides an additional 1.01% improvement at the block level. We open-source the entire training pipeline, including data collection, cleaning, three-stage training, and model inference, offering a validated framework to support code tasks on future new programming languages.