Low-Resource Project-Specific Code Summarizaiton: On the Importance of Datasets
摘要
Code-related pre-trained language models have achieved state-of-the-art performance in code summarization, and these models are trained and evaluated on large cross-domain datasets. But in the real-world software development scenarios, people are more concerned with generating project-specific summaries for the projects they are working on. While these models perform well in cross-domain datasets, our experiments reveal that they struggle to adapt to specific projects. Notably, data from the target project has a significant impact on improving performance. Unfortunately, many projects, particularly in their early stages, lack sufficient data for effective fine-tuning. In scenarios where project-specific data is limited, using data from projects within the same domain could be a potential solution. However, this approach has not been explored in prior work. This paper further investigates whether domain-specific data can improve model performance for a target project. Additionally, we also explore the effects of a second pre-training phase using different datasets on model performance. Our results show that training with a small amount of domain-specific data can yield comparable performance to models trained with large cross-project datasets. Moreover, a second pre-training phase further enhances the model’s performance on target projects.