LACD-MoE: Layer-Aware Adaptive Routing with Confidence-Difference Constraints for MoE Models
摘要
Over recent years, Mixture-of-Experts (MoE) models have emerged as a prominent research focus. In comparison to conventional dense large language models (LLMs), MoE architectures deliver superior performance while employing a reduced number of active parameters. Nevertheless, the efficient and precise allocation of experts to individual tokens within the MoE layer remains an enduring challenge in the field. To address this challenge, we propose LACD-MoE, which incorporates a layer-aware adaptive routing strategy alongside a confidence-difference constraint, improving model performance and enhancing the efficiency of computational resource utilization. In contrast to prior routing methods that activate a fixed number of experts, LACD-MoE dynamically adjusts the number of activated experts based on token-specific routing decisions. This adaptive strategy enables more efficient allocation of computational resources based on the complexity of individual tokens. Compared to existing dynamic routing methods, our approach explicitly models the relationship between expert activation and model depth, while incorporating a confidence-difference constraint to enable more precise and efficient expert selection. Extensive experiments across multiple datasets show that LACD-MoE outperforms state-of-the-art approaches by an average margin of 4.41%–8.50%, while also enhancing computational efficiency.