How Similar are Programming Solutions Generated by LLM-Based Generative AI Tools: A Semantic Clone Detection Approach
摘要
In recent days, a huge surge has been seen in the wide use of AI-powered chatbots for several tasks, such as interactive conversations, text generation, language translation, and many more. These chatbots, or Generative AI assistants, can also assist in generating code for programming problems. In this paper, we analyze the similarity of answers generated by two Generative AI assistants, ChatGPT (GPT 3.5) and YouChat (GPT 4), on different days. These assistants utilize Large Language AI Models (LLMs), which generate answers based on a probability distribution, and hence can generate different answers given the same problem. Thirty programming problems were selected from the popular coding website LeetCode. We instructed these chatbots to generate programs using Java as the programming language. The correctness of the answers generated by these LLMs is verified using Leetcode. Only those programming problems for which these LLMs could provide correct answers were considered for further analysis. To demonstrate the similarity among programs, we have used a clone detection technique proposed in our previous paper to find semantic clones (Gupta and Goyal in IEEE Access 12:70773–70791, 2024). The analysis of the resulting clone pairs shows that the generated programs contain approximately 76% of syntactic clones and 24% of semantic clones. Thus, these LLM-based Generative AI assistants depict a human-like ability to generate different answers to the same problems. Additionally, we used machine learning techniques to evaluate the ability of similarity scores between clone pairs’ documentations, bodies, and metrics to predict whether a clone is syntactic or semantic. Our findings indicate that these similarity scores can predict the type of clone with 80% accuracy.