Abstractive Summarization Evaluation for Prompt Engineering

The task of summarizing large documents for easier and faster readability is widely acknowledged and a standard task in the field of Natural Language Processing. The metrics that are used to measure the working of this task are based on statistical measures such as n-grams and Longest Common Subsequences. Abstractive summarization is a type of automatic text summarization which refers to creating the summary from the main document without entirely copying words from the original document. With the advent of Deep learning architectures, abstract summarization has increased in popularity. The Large language models which are used for abstractive text summarization need proper prompts to generate summary. Engineering proper prompts is important as the quality of the summary generated depends on the prompt. In this paper, an abstractive measure of similarity is proposed where the textual similarity is measured by using Euclidean distance to compare a Principal Component based transformed BERT Embedding vector of the document and the summary. The metric is used to create prompts for Generative Pretrained Transformer and Text to Text Transfer Transformer models which are standard state-of-the-art language models. The summary generated shows significant improvement and the prompts generated by using the abstractive comparison metric are seen to have the perplexity almost the same as the document thus promising better summarization results.