Assessing GPT-4 Generated Abstracts: Text Relevance and Detectors Based on Faithfulness, Expressiveness, and Elegance Principle

In recent years, the advancement of Artificial Intelligence (AI) technology has brought both convenience and panic. One of the most notable AI systems in recent years was ChatGPT in 2022. In 2023, GPT-4 was released as the latest version. Scholars are increasingly investigating the potential of ChatGPT/GPT-4 for text generation and summarization. Inspired by the principle of “Faithfulness, Expressiveness, and Elegance” in translation, this study investigates the writing and summarizing capabilities of GPT-4, one of the latest AI chatbots. For this purpose, we collected 60 articles from top financial and technology journals, extracted the abstract part, and fed it into GPT-4 to generate abstracts. Three evaluation metrics were created for evaluation: the Text Relevance Score, the AI Detector Score, and the Plagiarism Detector Score. Our findings indicate that abstracts generated by GPT-4 closely resemble the original abstracts without being detected by the plagiarism detector Turnitin in most cases. This implies that GPT-4 can produce logical and reasonable abstracts of articles on its own. Also, we conducted a cross-temporal analysis of GPT-4’s effectiveness and observed continuous and significant improvement. Nevertheless, with the advancement of AI detectors, the abstracts generated by GPT-4 can broadly be recognized as AI-generated. Furthermore, this paper also discusses ethical concerns and future research directions.