The Mixture-of-Experts (MoE) architecture is now widely adopted in the field of vision-language models due to its ability to significantly enhance model scale and efficiency. However, the internal operational dynamics of these complex systems, specifically the functional roles, redundancy levels, and critical interplay between the specialized routed and generalized shared expert pathways, remain largely uncharacterized. To probe this, we conduct a systematic study in a vision-language MoE using controlled routing restrictions and internal state analysis. We quantify shared expert contribution and identify substantial routed expert redundancy via generation quality assessment. Critically, we demonstrate a layer-dependent interplay where restricting routed experts decreases shared expert content stability while inducing compensatory increases in output magnitude, especially in early layers. These findings illuminate the coupled, adaptive nature of MoE components, informing model compression and design.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

How Do Shared Experts Dynamically Adapt to Routing Constraints in Mixture-of-Experts?

  • Huayang Li,
  • Jingjie Zeng,
  • Shaowu Zhang,
  • Liang Yang,
  • Yuanyuan Sun,
  • Kan Xu,
  • Hongfei Lin

摘要

The Mixture-of-Experts (MoE) architecture is now widely adopted in the field of vision-language models due to its ability to significantly enhance model scale and efficiency. However, the internal operational dynamics of these complex systems, specifically the functional roles, redundancy levels, and critical interplay between the specialized routed and generalized shared expert pathways, remain largely uncharacterized. To probe this, we conduct a systematic study in a vision-language MoE using controlled routing restrictions and internal state analysis. We quantify shared expert contribution and identify substantial routed expert redundancy via generation quality assessment. Critically, we demonstrate a layer-dependent interplay where restricting routed experts decreases shared expert content stability while inducing compensatory increases in output magnitude, especially in early layers. These findings illuminate the coupled, adaptive nature of MoE components, informing model compression and design.