Scholarlink

<p>Mixture of Experts (MoE) models reduce computational costs through a sparse activation strategy, making it widely adopted in large-scale model training. However, MoE structures still face issues such as knowledge redundancy and token overflow caused by static routing. Existing solutions typically involve increasing expert capacity or applying stricter loss constraints to balance expert loads. These approaches can lead to wasted computational resources due to complex loss functions and free expert capacity. This paper proposes an optimization strategy that combines consensus experts with dynamic capacity allocation. Firstly, in the MoE structure of the consensus expert, the <b>ARIMA</b> time series prediction algorithm is used to predict future load distribution, and the experts’ dynamic capacity is allocated to alleviate routing fluctuations during the initial training phase. Secondly, by comparing the predicted load with the actual load, the loss function is reconstructed to avoid pursuing absolute balance. Experiments on the <b>Switch Transformer-base-8</b> model show that our method achieves an average accuracy of 81.55% across 9 GLUE benchmark tasks—an improvement of 1.07% over the baseline. The gains are notable on imbalanced datasets (e.g., MRPC), with improvements of up to 1.8%. Further experiments demonstrate the method’s effectiveness at expert numbers of 16 and 32.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Consensus-Expert DynamicMoE: ARIMA-based Capacity Prediction with Adaptive Load Balancing for Sparse Models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月12日

查看全文
  • Jia-Lin Wen,
  • Xiao-Jun Li,
  • Jun-Ping Yao,
  • Hai-Feng Sun,
  • Xiang-Rui An

摘要

Mixture of Experts (MoE) models reduce computational costs through a sparse activation strategy, making it widely adopted in large-scale model training. However, MoE structures still face issues such as knowledge redundancy and token overflow caused by static routing. Existing solutions typically involve increasing expert capacity or applying stricter loss constraints to balance expert loads. These approaches can lead to wasted computational resources due to complex loss functions and free expert capacity. This paper proposes an optimization strategy that combines consensus experts with dynamic capacity allocation. Firstly, in the MoE structure of the consensus expert, the ARIMA time series prediction algorithm is used to predict future load distribution, and the experts’ dynamic capacity is allocated to alleviate routing fluctuations during the initial training phase. Secondly, by comparing the predicted load with the actual load, the loss function is reconstructed to avoid pursuing absolute balance. Experiments on the Switch Transformer-base-8 model show that our method achieves an average accuracy of 81.55% across 9 GLUE benchmark tasks—an improvement of 1.07% over the baseline. The gains are notable on imbalanced datasets (e.g., MRPC), with improvements of up to 1.8%. Further experiments demonstrate the method’s effectiveness at expert numbers of 16 and 32.