Enhancing pre-trained models for text summarization: a multi-objective genetic algorithm optimization approach

Pre-trained language models have significantly advanced text summarization by leveraging extensive pre-training data to enhance performance. Many cutting-edge models undergo an initial pre-training phase on a large corpus before being fine-tuned specifically for text summarization tasks. The challenge arises when the limited data available for these summarization tasks and the high complexity of the models lead to aggressive fine-tuning, which can cause the models to over fit the training data and fail to generalize well to new, unseen data. To address these challenges, the study proposes a multi-objective based Genetic Algorithm (MOGA) approach for optimizing pre-trained models for summarization The proposed approach first compares the performance of various pre-trained models by fine-tuning them with the CNN/Daily Mail dataset. Secondly, the proposed Multi-Objective Genetic Algorithm (MOGA) is applied to all validated pre-trained models and compared using a statistical approach. Experimental results reveal that Pre-training with Extracted Gap-sentences for Abstractive Summarization Sequence-to-sequence (PEGASUS) model performs best among the pre-trained models. Further optimization with the proposed MOGA method yields even better results, with the optimized model achieving a Recall-Oriented Understudy for Gisting Evaluation) ROUGE-1 score of 0.514, a ROUGE-2 score of 0.4, and a ROUGE-L score of 0.5.