Multi-level context Modeling for consistent expert selection in Mixture-of-Experts

2026-07-17

AI Summary

This work addresses the limitations of existing Mixture-of-Experts (MoE) models, whose expert routing relies on shallow or isolated token representations, leading to unstable and semantically inconsistent decisions. To overcome this, the authors propose the MCF-MoE framework, which, for the first time, approaches routing from the perspective of representational completeness. By integrating cross-layer semantic aggregation and local token interaction modeling, MCF-MoE constructs context-aware, multi-granular representations that enable more consistent and informative expert selection. The method substantially improves both routing consistency and expert specialization, outperforming strong MoE baselines across language modeling and understanding benchmarks, thereby demonstrating the critical role of contextual completeness in effective expert routing.

Abstract

Mixture-of-Experts (MoE) enables efficient scaling of Transformer models by routing tokens to a small subset of experts. However, existing routers typically condition expert selection on shallow or isolated token representations, which often produce unstable and semantically inconsistent routing decisions across layers. In this work, we revisit expert selection from a representation perspective and identify context incompleteness as a key bottleneck limiting effective expert specialization. To address this issue, we propose Multi-level Context Fusion MOE (MCF-MOE), a framework that constructs context-aware representations by integrating complementary signals from cross-layer semantic aggregation and local token-level interactions, enabling more informative and consistent expert selection. Experiments on language modeling and understanding benchmarks demonstrate that MCF-MOE consistently improves routing consistency and downstream performance over strong MoE baselines, highlighting the importance of contextual completeness in expert routing. The code is available at https://anonymous.4open.science/r/MCFMOE.

Problem

• Mixture-of-Experts
• expert selection
• context incompleteness
• routing consistency
• semantic inconsistency

Innovation

• Mixture-of-Experts
• context-aware routing
• cross-layer aggregation
• expert selection consistency
• context completeness