This paper is an attempt to automate the evaluation of the performance of an automatic summarization system by comparing the golden summary with the predicted summary. The paper proposes two methods for evaluation and both of them score the predicted summary based on how semantically similar it is to its corresponding golden summary. The first method uses S-BERT’s pre-trained sentence embeddings to generate sentence feature vectors for the predicted and golden summary correspondingly. Cosine similarity is used to determine the similarity between the two sentence Vectors. The second method uses Singular Value Decomposition (SVD) on the embeddings of both the gold and predicted summary to generate the latent topics. Angular cosine similarity is computed between the corresponding vectors of the gold and the predicted summary, taking into consideration only the top 2 latent topics.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Automatic Evaluation of Text Summaries

  • Aiom Minnette Mitri,
  • N. Donald Jefferson Thabah,
  • Arnab Kumar Maji,
  • Goutam Saha,
  • Bipul Shyam Purkayastha

摘要

This paper is an attempt to automate the evaluation of the performance of an automatic summarization system by comparing the golden summary with the predicted summary. The paper proposes two methods for evaluation and both of them score the predicted summary based on how semantically similar it is to its corresponding golden summary. The first method uses S-BERT’s pre-trained sentence embeddings to generate sentence feature vectors for the predicted and golden summary correspondingly. Cosine similarity is used to determine the similarity between the two sentence Vectors. The second method uses Singular Value Decomposition (SVD) on the embeddings of both the gold and predicted summary to generate the latent topics. Angular cosine similarity is computed between the corresponding vectors of the gold and the predicted summary, taking into consideration only the top 2 latent topics.