Performance Evaluation of Text Summarization Models on SAMSUM Chat Data
摘要
There is widespread dependence on messaging apps and automated chatbots in various situations. After a long debates, individuals may need to review the discussion’s main points. Different approaches for extractive and abstractive text summarizing are carried out in this article to answer the demand for text summarization in messaging systems. This paper implements the TF-IDF method for extractive summarization and the abstractive summarization system which uses transformer-based models from the hugging face, such as T5, BART, and PEGASUS. These transformer-based models are trained on the SAMSUM dataset with 3000 instances of hugging face using the Auto Train facility. The article evaluates different summarization approaches based on the ROUGE score obtained from testing the models on 100 test instances of the dataset. The experiment results show that the ROUGE scores obtained using abstractive methods are better than those obtained using extractive methods, as the human-generated summaries in the SAMSUM dataset are more abstractive. The PEGASUS model gets the best ROUGE scores, but other models are also of comparable quality. Experimental analysis reveals that PEGASUS, BART, and T5 outperform TF-IDF method with a ROUGE-1 score of 47.90, compared to 43.84 and 44.43, respectively. In contrast, the TF-IDF method achieved a substantially lower ROUGE-1 score of 19.31. Similar trends are observed for ROUGE-2 and ROUGE-L metrics. Notably, the precision value of ROUGE-2 for TF-IDF is 36.48, which, although lower than the precision values of the BART—64.15, T5—39.81, PEGASUS—43.51.