In Open-domain Question Answering (OpenQA), retrievers play a critical role in extracting relevant passages from massive text corpora, whose effectiveness heavily depends on the quality of training data. While existing studies mitigate this dependency via soft-label generation, their reliance on single-likelihood-based relevance scoring and coarse-grained document ranking leads to poor discrimination of hard samples and fails to capture fine-grained relevance differences between documents. To address these issues, we propose Joint Likelihood-based Soft-label Generation (JointLSG), a novel strategy that improves relevance modeling by leveraging the joint likelihood of question and answer generation, enabling finer-grained analysis. We first systematically compare soft-label construction methods derived from varying posterior probabilities, revealing the complementary nature of answer and question generation strategies. By integrating their joint probabilities, our derived soft-labels retain fine-grained relevance information while substantially enhancing scoring accuracy. Experimental results demonstrate that JointLSG-generated soft-labels achieve significantly better direct ranking performance than UPR and RankGPT across the MSMARCO, Natural Questions, and TriviaQA datasets, outperforming UPR by up to 28.72% and even surpassing RankGPT by 25.02%. Furthermore, on the DL19 and DL20 test sets, the retriever trained with JointLSG exhibits a clear advantage over other soft-labeling strategies, offering a highly accurate and resource-efficient solution for open-domain QA systems.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Supporting Retriever’s Training by Joint Likelihood-Based Soft-Label Generation

  • Tailai Peng,
  • Rui Chen,
  • Xinran Xie,
  • Dekun Lin,
  • Zhe Cui

摘要

In Open-domain Question Answering (OpenQA), retrievers play a critical role in extracting relevant passages from massive text corpora, whose effectiveness heavily depends on the quality of training data. While existing studies mitigate this dependency via soft-label generation, their reliance on single-likelihood-based relevance scoring and coarse-grained document ranking leads to poor discrimination of hard samples and fails to capture fine-grained relevance differences between documents. To address these issues, we propose Joint Likelihood-based Soft-label Generation (JointLSG), a novel strategy that improves relevance modeling by leveraging the joint likelihood of question and answer generation, enabling finer-grained analysis. We first systematically compare soft-label construction methods derived from varying posterior probabilities, revealing the complementary nature of answer and question generation strategies. By integrating their joint probabilities, our derived soft-labels retain fine-grained relevance information while substantially enhancing scoring accuracy. Experimental results demonstrate that JointLSG-generated soft-labels achieve significantly better direct ranking performance than UPR and RankGPT across the MSMARCO, Natural Questions, and TriviaQA datasets, outperforming UPR by up to 28.72% and even surpassing RankGPT by 25.02%. Furthermore, on the DL19 and DL20 test sets, the retriever trained with JointLSG exhibits a clear advantage over other soft-labeling strategies, offering a highly accurate and resource-efficient solution for open-domain QA systems.