Improve Conversational Search with Multi-document Information

Conversational Search (CS) aims to satisfy complex information needs via multi-turn user-agent interactions. During this process, multiple documents need to be retrieved based on the conversation history to respond to the user. However, existing approaches still make it difficult to distinguish irrelevant information from the user’s question at the semantic level. When a conversation involves multiple documents, it sometimes affects the retrieval performance negatively. In order to enhance the model’s ability to comprehend conversations and distinguish passages in irrelevant documents via multi-document information, we propose an unsupervised multi-document conversation segmentation method and a zero-shot Large Language Model (LLM)-based document summarization method to extract multi-document information from conversation history and documents respectively for amending the lack of training data for extracting multiple document information. We further present the Passage-Segment-Document (PSD) post-training method to train the reranker using the extracted multi-document information in combination with a multi-task learning method. The results on the MultiDoc2Dial dataset verifies the improvement of our method on retrieval performance. Extensive experiments show the strong performance of our method for dealing with conversation histories that contain multi-document information.