Evaluating few-shot and contrastive learning methods for code clone detection

Code Clone Detection (CCD) is a software engineering task that is used for plagiarism detection, code search, and code comprehension. Recently, deep learning-based models have achieved an F1-Score (a metric used to assess classifiers) of \(\sim \) $\sim$ 95% on the CodeXGLUE benchmark. These models require many training data, mainly fine-tuned on Java or C++ datasets. However, no previous study evaluates the generalizability of these models where a limited amount of annotated data is available.

The main objective of this research is to assess the ability of the CCD models as well as few-shot learning algorithms for unseen programming problems and new languages (i.e., the model is not trained on these problems/languages).

We assess the generalizability of the state-of-the-art models for CCD in few-shot settings (i.e., only a few samples are available for fine-tuning) by setting three scenarios: i) unseen problems, ii) unseen languages, iii) combination of new languages and new problems. We choose CodeNet and conduct our experiments on Java, C++, and Ruby languages. Then, we employ Model Agnostic Meta-learning (MAML), where the model learns a meta-learner capable of extracting transferable knowledge from the train set; so that the model can be fine-tuned using a few samples. Finally, we combine contrastive learning with MAML to further study whether it can improve the results of MAML.

Our results show that the performance of the models drops \(\sim 50\%\) $\sim 50\%$ for Java and \(\sim 20\%\) $\sim 20\%$ for C++ and Ruby for unseen problems, which are then boosted by \(13\%\) $13\%$ to \(24\%\) $24\%$ F1 scores for Java and C++/Ruby, respectively when MAML is used. Similar observations are found for unseen languages and the third scenario. Though in case of third scenario (i.e., unseen problems and unseen languages) the scores are lower. Integrating contrastive learning with MAML did not help in boosting the performance more than what we could achieve with MAML. Our results open new avenues of research and the need to develop robust models for clone detection, in the settings we investigated here.