AI-generated or AI touch-up? Identifying AI contribution in text data

Since the advent of ChatGPT, detecting AI-generated text to address the rising concerns about the authenticity and potential misuse of AI-generated text has been the focus of many researchers. While previous studies have mainly focused on distinguishing between AI-generated text and human-written text, our research aims to further explore identifying different uses of a generative model. In a scenario in which a student uses AI to enhance the clarity and coherence of their assignments, existing detection tools often label such enhancements as entirely AI generated, raising doubts about the authenticity of the student’s work. To address this, we propose AICIS-2S, a two-step AI Contribution Identification System that classifies text into three categories: human-written, AI-enhanced (human-written text polished by AI), or AI-originated (fully generated by AI). For our experiments and evaluations, we created a dataset of 41,191 records, encompassing samples belonging to each of these three classes, collected from an online question-answering platform and generated from ChatGPT (GPT-3.5). In our model design, we prioritized minimizing false positives to avoid mistakenly questioning the authenticity of human authors. Our experimental results on the provided dataset demonstrate AICIS-2S’s ability to achieve a precision of 99.27% for the AI-originated class while maintaining a high overall accuracy of 98.83%. Additionally, we compared AICIS-2S against GPTZero, a leading AI detection tool, revealing GPTZero’s tendency to misclassify AI-enhanced samples as AI-originated. By addressing this limitation, AICIS-2S offers a promising solution to preserve text authenticity in various domains, such as education and academia.