Too Much of a Good Thing: When sim2real Efforts Impede Policy Learning (And What to Do About It)

2026-05-30

AI Summary

This work addresses the limitations of conventional sim2real transfer, where excessive alignment between simulation and reality often restricts policy learning, hampers exploration, and leads to simulator overfitting. To overcome these issues, the authors propose a novel “sim2sim2real” paradigm that leverages only robotic kinematics as a constraint, thereby preserving real-world feasibility while substantially enhancing policy flexibility and exploratory capacity. By integrating a lightweight transfer framework with purposefully designed simulation environments, the approach effectively mitigates simulator locking and significantly improves both generalization performance and training efficiency when deploying policies on real hardware.

Abstract

While sim2real efforts are necessary for effective policy transfer to hardware, there is such a thing as too much of a good thing. We argue that sim2real efforts have led to misaligned incentives with policy learning, resulting in simulator lock in and poor policy exploration due to the unreasonable constraints imposed by the real world. We offer a diagnosis and explanation of the current status of the problem, and propose a potential solution via a sim2sim2real paradigm that leverages the robot's kinematics as the sole design constraint.

Problem

• sim2real
• policy learning
• simulator lock-in
• policy exploration
• real-world constraints

Innovation

• sim2real
• policy learning
• simulator lock-in
• sim2sim2real
• kinematics constraint