A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control

2026-10-08

AI Summary

This study addresses the low exploration efficiency and substantial engineering overhead caused by uniform sampling in large-scale reinforcement learning by proposing a capability-boundary-oriented adaptive sampling mechanism. This method dynamically focuses on task configurations at the capability frontier within million-scale parallel simulation environments, overcoming ultra-large-scale exploration bottlenecks and significantly improving the utilization of learning signals. Furthermore, by integrating sim-to-real reinforcement learning with visual policy distillation techniques, this work achieves zero-shot transfer to physical robots for both legged locomotion over complex terrains and precision assembly tasks. These results effectively resolve the generalization challenges that remain difficult for conventional approaches.

Abstract

General-purpose robots must perform a wide range of tasks from agile locomotion to dexterous manipulation. While sim-to-real reinforcement learning (RL) has proven to be a useful tool for this goal, current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations. Recent work has shown that diverse simulator resets, combined with massively parallel simulation, can alleviate much of this engineering burden on several manipulation problems. However, we find that naively scaling this paradigm to more precise or dynamic problems remains non-trivial. While simulator resets can help with exploration, uniformly sampling over this distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt. This makes it challenging to see the expected benefits of scaling parallel environments for RL, since much of the learning signal in a batch is wasted during learning. To mitigate this, we introduce Success Guided Sampling (SGS), a simple adaptive sampler that concentrates RL training on task configurations around the frontier of the policy's capabilities. Doing so allows large-scale simulated RL to make the most out of the experience in a batch, enabling much more effective scaling to large-scale parallel simulation. Across experiments using up to $2^{20}$ (over one million) parallel environments, SGS enables RL to solve challenging multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve. Finally, we distill the learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware. Project website: https://sgs-rl.github.io/.

Problem

• Reinforcement Learning
• Robot Control
• Exploration Bottleneck
• Sim-to-Real
• Mega-Scale Parallel Simulation

Innovation

• Success Guided Sampling
• Reinforcement Learning
• Massively Parallel Simulation
• Sim-to-Real Transfer
• Robot Control