IR Embedding Fairness Inspection via Contrastive Learning and Human-AI Collaborative Intelligence

Xiaohongshu’s search daily serves tens of millions active users in social networks, that presents a challenge to existing log-based embedding based retrieval (EBR) system: how to endorse individual document exposure fairness to diversify the search results. Conventional EBR models optimize relevance between query and document by leveraging massive user behavior data, e.g. clicks, purchase, etc., however, search log derived retrieval outcomes can deviate from true relevance distribution, that may result in less opportunity to retrieve for low-popularity or long-tailed documents. To address this problem, in this study, we propose a novel semi-supervised model, Gaussian process based contrastive learning (GPCL), which minimizes the discrepancy between model prediction distribution and true relevance distribution via taking advantage of contrastive samples adaptively generated from small human-labeled data. We validated the effectiveness of the proposed methodology by comparing with a set of baselines and observed significant metrics gains via online A/B testing. We discuss the entire system including model deployment and parameter-tuning. Also the new dataset is publicly available, which associates manually labeled relevance samples and massive click-logs.