Advancing Abstractive Summarization: Evaluating GPT-2, BART, T5-Small, and Pegasus Models with Baseline in ROUGE and BLEU Metrics

An intriguing problem in natural language processing is text summarising, which requires extracting important information from a text to create concise and informative summaries. Advancements in the domain, especially with the use of Transformer-based models, have produced outstanding results on a variety of benchmarks. Using the CNN/Daily Mail dataset, this study carefully assesses four important text summarising algorithms: GPT-2, BART, T5-Small, and Pegasus. As a baseline model or human-written summaries, we start by taking into account the first three lines of each article in the dataset. The dataset is used to train all four models, including GPT-2, a large language model constructed on a Transformer architecture. They have the job of summarising, and the results are kept in a dictionary called "summaries". Each model then produces summaries when it is fed fresh input text from the dataset’s article column. The resulting BLEU and ROUGE scores allow for a comparison examination to determine which model performs best. A variety of visualisation methods, including bar graphs and heat maps, improve the evaluation of model performance. ROUGE scores offer numerical assessments of summary quality, whereas mean, median, and standard deviation offer perceptions into central patterns and variations among many assessment metrics. Comparing the summaries that our algorithms produce with those that are already in the dataset is a crucial step. We can determine whether model—including the baseline—produces summaries similar to those in the highlights column by comparing it to this comparison. The fact that Pegasus consistently outperforms the other models is noteworthy. This observation is consistent with its effectiveness in text summarization across a range of assessment parameters. The larger picture, however, shows that despite Transformer models’ enormous promise in this field, difficulties still exist. Factual correctness, thoroughness, and flow in produced summaries are among the areas that continue to be the focus of research and improvement. Finally, a practical test is conducted to see how the models react after inputting specific input. Their performance outside of the training dataset is robustly validated by this real-world application. Pegasus stands out as a front-runner due to its continuous performance. The study not only demonstrates the improvements in text summarization made possible by Transformer models, but it also emphasises the necessity of ongoing research and development in order to overcome current obstacles.