Loop the Loopies!

2026-07-17

AI Summary

This work addresses the performance bottleneck in recurrent Transformers, where scaling parameters outperforms increasing recurrence depth under growing pretraining compute budgets. To overcome this limitation, the authors introduce the Loopie family of models, which integrates recurrent Transformer architectures with a Mixture-of-Experts (MoE) design and employs a novel, efficient post-training pipeline. Under identical computational budgets, Loopie significantly surpasses standard Transformer baselines. Experimental results demonstrate that its sparsely activated configurations—20B/2B and 6B/0.6B—achieve gold-medal performance on the 2025 International Mathematical Olympiad (IMO) and International Physics Olympiad (IPhO) without external tool assistance, substantially outperforming a 30B dense model. This marks the first instance where a recurrent architecture exceeds pure parameter scaling under equivalent computational expenditure.

Abstract

We present Loopie, the most powerful looped Transformer to date. The Loopie series consists of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6Bparameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N-fold increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. Our novel post-training pipeline equips Loopie with strong reasoning abilities. At the 2025 IMO and IPhO, Loopie achieves gold-medal performance without tools.

Problem

• Looped Transformers
• Mixture-of-Experts
• parameter scaling
• compute efficiency
• model looping

Innovation

• Looped Transformer
• Mixture-of-Experts
• post-training pipeline
• reasoning ability
• compute-efficient scaling