How Do Shared Experts Dynamically Adapt to Routing Constraints in Mixture-of-Experts?
摘要
The Mixture-of-Experts (MoE) architecture is now widely adopted in the field of vision-language models due to its ability to significantly enhance model scale and efficiency. However, the internal operational dynamics of these complex systems, specifically the functional roles, redundancy levels, and critical interplay between the specialized routed and generalized shared expert pathways, remain largely uncharacterized. To probe this, we conduct a systematic study in a vision-language MoE using controlled routing restrictions and internal state analysis. We quantify shared expert contribution and identify substantial routed expert redundancy via generation quality assessment. Critically, we demonstrate a layer-dependent interplay where restricting routed experts decreases shared expert content stability while inducing compensatory increases in output magnitude, especially in early layers. These findings illuminate the coupled, adaptive nature of MoE components, informing model compression and design.