Multi-document summarization (MDS) is a technique for automatically generating a concise summary of information from multiple sources related to a specific topic. Current evaluation methods for multi-document summarization (MDS) rely on metrics like ROUGE scores, which don’t reflect real-world use cases. This study proposes a novel query-based approach that assesses how well summaries address user information needs. This method aligns better with the practical application of MDS in the age of information overload. We leverage advanced abstractive summarization models, including BERT, GPT-2, Pegasus, and others, to generate summaries. Our evaluation focuses on whether these summaries effectively answer user-defined queries. This shift from generic metrics to a user-centric approach aims to identify the most effective MDS model by ensuring summaries are not only coherent but also directly address user information needs. This ultimately improves the accuracy and practical value of multi-document summarization systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Query-Based Evaluation of Multi-document Summarization Models

  • Raji Ramachandran,
  • Christy Mariya,
  • A. S. Shini

摘要

Multi-document summarization (MDS) is a technique for automatically generating a concise summary of information from multiple sources related to a specific topic. Current evaluation methods for multi-document summarization (MDS) rely on metrics like ROUGE scores, which don’t reflect real-world use cases. This study proposes a novel query-based approach that assesses how well summaries address user information needs. This method aligns better with the practical application of MDS in the age of information overload. We leverage advanced abstractive summarization models, including BERT, GPT-2, Pegasus, and others, to generate summaries. Our evaluation focuses on whether these summaries effectively answer user-defined queries. This shift from generic metrics to a user-centric approach aims to identify the most effective MDS model by ensuring summaries are not only coherent but also directly address user information needs. This ultimately improves the accuracy and practical value of multi-document summarization systems.