The Performance of GPT-3.5 in Summarizing Scientific and News Articles

In the age of information, we are overwhelmed with large amounts of data. The quest to know more in less time has increased the need for efficient text summarization models that convert information into precise summaries such that essential details are not overlooked. Recently, GPT-3.5 has demonstrated impressive performance in text completion, generation, and question answering. However, its effectiveness in generating concise and coherent summaries for scientific articles and news reports remains under-explored. This work evaluates the performance of GPT-3.5 in summarizing scientific research articles and news data. Scientific articles were collected from arXiv STEM dataset, whereas news articles were sampled from the CNN/DailyMail dataset. Using the GPT-3.5 OpenAI API, the pre-trained model is prompted to generate summaries of the scientific and news articles. In the next step, the ROUGE score is computed for the generated summaries against the reference summaries to analyse the performance of the model. Our results show that GPT-3.5 performs slightly better in summarizing scientific articles as compared to news articles with an average ROUGE score of 0.35 and 0.31, respectively. Moreover, in agreement with the literature, we show that the ROUGE is not the best measure for evaluating text similarity as it heavily relies on similar vocabulary rather than semantics.