PNRM: A Pre-training News Recommendation Model with Contrastive Sampling

With the swift rise of online news recommendation platforms, people can access the latest news quickly and conveniently. News recommendation models are central to these systems, enabling the inference of user interests and the delivery of accurate suggestions. Many existing systems adopt supervised learning based on user behavior records. However, user feedback is typically noisy and insufficient, ultimately impairing the performance of news recommendation models. To alleviate this problem, in this paper, we propose PNRM, a news recommendation framework that integrates self-supervised contrastive pre-training. We explore intrinsic signals from the raw interaction data, alleviate the data sparsity problem, and obtain more robust representations from noisy user behaviors. PNRM follows a two-stage training paradigm comprising pre-training and fine-tuning. (1) In the pre-training stage, we design three contrastive learning tasks, namely user-user contrast, user-news contrast, and news-news contrast, to mine intrinsic supervisory signals from large-scale unlabeled interactions. (2) In the fine-tuning stage, the news encoder and the user encoder are initialized with the parameters from the pre-training stage and further fine-tuned with the click data. This two-stage pipeline first acquires robust representations using self-supervision, then fine-tunes them for recommendation, equipping the model with stronger generalization and task-specific performance. Experiments conducted on Microsoft News Dataset (MIND) substantiate that our proposed method outperforms the basic model of a strong BERT-enhanced two-tower baseline, which leads publicly available two-tower baselines on MIND, showcasing the effectiveness of our proposed method.