All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learning

2026-05-31

AI Summary

This work addresses a critical challenge in model-based reinforcement learning (MBRL)—the tendency of agents to over-rely on inaccurate dynamics models, which undermines both learning efficiency and safety. To mitigate this issue, the paper introduces an uncertainty-aware modeling mechanism that explicitly quantifies epistemic uncertainty within probabilistic dynamics models, thereby constructing a misuse-resistant learning framework. By accounting for model uncertainty during policy optimization, the proposed approach effectively alleviates performance degradation caused by model errors. Empirical evaluations on real robotic platforms demonstrate that the method enables efficient and safe direct learning and exploration, substantially improving data efficiency and safety in MBRL. This advancement contributes significantly to the development of uncertainty-aware MBRL systems capable of reliable deployment in real-world environments.

Abstract

Model-based reinforcement learning (MBRL) infers information about the environment from a learned dynamics model and bears the potential to address open problems such as data efficient and safe learning in robotics. However, inaccuracies of the learned dynamics model are typically exploited by the agent, substantially hampering the capabilities of MBRL methods. We present a framework for dealing with inaccuracies of probabilistic models through targeted handling of uncertainty that effectively mitigates model exploitation. We present recent successes in learning directly on hardware and safe exploration, and discuss future directions for uncertainty-aware MBRL.

Problem

• model uncertainty
• reinforcement learning
• model-based reinforcement learning
• dynamics model inaccuracies
• safe learning

Innovation

• model uncertainty
• model-based reinforcement learning
• safe exploration
• probabilistic dynamics model
• uncertainty-aware learning