Query-Based Evaluation of Multi-document Summarization Models
摘要
Multi-document summarization (MDS) is a technique for automatically generating a concise summary of information from multiple sources related to a specific topic. Current evaluation methods for multi-document summarization (MDS) rely on metrics like ROUGE scores, which don’t reflect real-world use cases. This study proposes a novel query-based approach that assesses how well summaries address user information needs. This method aligns better with the practical application of MDS in the age of information overload. We leverage advanced abstractive summarization models, including BERT, GPT-2, Pegasus, and others, to generate summaries. Our evaluation focuses on whether these summaries effectively answer user-defined queries. This shift from generic metrics to a user-centric approach aims to identify the most effective MDS model by ensuring summaries are not only coherent but also directly address user information needs. This ultimately improves the accuracy and practical value of multi-document summarization systems.