How to Loop MoE: Flatten the Experts, Untie the Attention

2026-09-28

AI Summary

This study addresses the challenges of low expert utilization and imbalanced routing decisions in recurrent Mixture-of-Experts (MoE) models by proposing Foil. This method introduces, for the first time, expert flattening and attention decoupling strategies to synergistically optimize recurrent architectures and width scaling benefits under a fixed parameter budget. Experimental results demonstrate that Foil significantly reduces pretraining loss while achieving comparable or superior accuracy on downstream tasks, alongside more confident and balanced dynamic routing. Furthermore, this work reveals the synergistic effects between recurrent Transformers and sparse MoE architectures, establishing a new paradigm for efficient dynamic routing mechanisms.

Abstract

Looped Transformers reuse one block of layers several times: by spending extra computation they push a model of fixed size further, and so use its parameters more fully; while sparse mixture-of-experts (MoE) models activate only a few of many experts for each token. Looped MoE bridges these two design philosophies and gives MoE models new potential for better expert usage, but it raises a question: how to loop a MoE? We answer it with Foil. With the expert parameters and the expert compute per token held fixed, Foil (1) flattens the experts, halving the expert layers, doubling the experts per layer and doubling the passes, so that every routing decision chooses from a larger pool, and (2) unties the attention, giving each pass its own attention parameters while the experts and routers stay shared. Experiments show that Foil clearly outperforms the unflattened looped baseline: at 20B tokens every Foil model has lower pretraining loss than the baseline; at 100B tokens the loss improves monotonically with the degree of flattening, the most flattened Foil ending 0.012 nat below the baseline at equal parameters and compute, with downstream accuracy on par or better; untying the attention also yields more balanced and more confident routing at equal shape. Our ablations analyse why Foil works and turn the findings into design guidance for looped MoE: the returns of looping and of widening the expert layers amplify each other, routing confidence tracks healthy expert use better than load balance, and a sparse looped MoE should therefore use more experts per layer and more passes. Code and configurations are available at https://github.com/SR-A-W/how-to-loop-moe.

Problem

• Looped Transformers
• Mixture-of-Experts
• Looped MoE
• parameter efficiency

Innovation

• Looped MoE
• Expert Flattening
• Untied Attention
• Sparse Mixture-of-Experts
• Routing Confidence