Improving Alignment and Uniformity of Expert Representation with Contrastive Learning for Mixture-of-Experts Model

Multi-task prediction in recommendation systems has garnered considerable attention, particularly with the success of Mixture of Experts (MoE) based models such as MMoE and PLE. In this paper, we first observed that many existing MoE-based models prioritize increasing model capacity for better online performance, resulting in redundant or similar expert hidden representations. This negatively impacts online performance and parameter utilization efficiency. To address this, we introduce self-supervised learning to improve alignment and uniformity of expert representation and propose a Contrastive Learning for MoE models (CMoE) framework consisting of two self-supervised learning signals: Experts Homogeneity Penalty(EHP), Expert Agreement Regularization (EAR). The EHP ensures distinct hidden representations for different experts, while the EAR enhances feature representation learning. We conducted experiments on real-world datasets for Click-Through Rate (CTR), Conversion Rate (CVR), and Deep Conversion Rate (DVR) prediction tasks. Results showed significant improvements, with a 1.24% increase in AUC compared to the baseline model. Online A/B tests also validated the approach, demonstrating a 3.27% enhancement in CTCVDVR and 3.42% in ARPU (Average Revenue Per User). The code has been available at https://github.com/BZX667/CMoE .