MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement

2026-10-08

AI Summary

This study addresses the challenge of scaling reinforcement learning (RL) compute to enhance the self-evolution capabilities of omni-modal foundation models. To this end, we propose a pretraining scheme based on a hybrid SWA architecture and construct a large-scale asynchronous RL infrastructure supporting million-token contexts. Furthermore, we introduce group-wise agent scoring to optimize reward signals and mitigate reward hacking by freezing MoE routing alongside multi-layer defense mechanisms. Our approach significantly improves model performance and token efficiency across complex multimodal tasks. By open-sourcing the training dynamics and environments, this work aims to facilitate broader community research in scalable RL for foundation models.

Abstract

Reinforcement learning (RL) is the central training paradigm for advancing large foundation models towards self-improvement. This report introduces the MiMo-V2.6 series, an omni-modal family that pushes the frontier of model intelligence by scaling RL compute. Prior to RL, we conduct mid-training on a broad multimodal corpus to provide ample exploration space, and build a solid infrastructure on the pretrained hybrid-SWA architecture to support subsequent scale-up. We scale RL compute along three dimensions: (1) larger batches and higher throughput, with an asynchronous training that consumes 1,568 samples and 2.7-3.7B tokens per step at context lengths of up to 1M; (2) more diverse and complex environments, spanning code, general, visual, and cyber domains under a mixture of agent harnesses; and (3) more grader compute, via groupwise agentic grading that yields more accurate reward signals for long-horizon tasks and steers the model towards shorter, more token-efficient solutions. To keep training stable at scale, we freeze the MoE router and establish a multi-layer defense against reward hacking. We further build infrastructure for mixed-task agentic RL, including a unified trajectory representation, high-concurrency multi-framework rollout, decoupled control and data planes, and training-inference consistency. We open-source the training dynamics, RL environments, and RL framework to facilitate reproduction and further research on scaled RL and model self-improvement.

Problem

• Reinforcement Learning
• Self-Improvement
• Omni-modal Foundation Models
• Scaling RL Compute
• Reward Hacking

Innovation

• Reinforcement Learning
• Omni-modal
• Agentic Grading
• Reward Hacking Defense
• Mixed-task Agentic RL