Beyond Conservatism: Recoverability-Conditioned Exploration for Model-Based Imitation Learning

2026-09-27

AI Summary

This study addresses the problem of insufficient out-of-distribution exploration caused by conservative policies in model-based imitation learning. To this end, it proposes the RECON framework, which employs a dual-policy architecture comprising a main policy and an explorer to decouple exploitation from exploration. Furthermore, the method introduces a recoverability condition based on multi-step imagination to guide data collection, ensuring that explored states can safely revert to expert behavior and thereby preventing futile exploration. By integrating epistemic uncertainty estimation with model-based reinforcement learning, the proposed approach significantly enhances sample efficiency, imitation performance, and robustness across locomotion, navigation, and manipulation tasks, consistently outperforming existing baseline methods.

Abstract

Model-based imitation learning (MBIL) improves real-environment interaction efficiency by optimizing policies on imagined rollouts from a learned world model. However, the gap between model-induced and real-environment occupancies makes policy learning sensitive to model error. Conservative MBIL mitigates model exploitation during policy optimization, but when real-environment interactions are collected by the same conservative policy, uncertain regions around the expert distribution remain insufficiently sampled. Generic uncertainty-driven exploration, on the other hand, may allocate interaction to novel but task-irrelevant dynamics. We propose REcoverability-CONditioned Exploration for Model-Based Imitation Learning (RECON). RECON separates conservative policy learning from active data collection by maintaining a main policy for task execution and an explorer for real-environment interaction. The explorer is optimized based on epistemic uncertainty conditioned on recoverability estimated from multi-step main-policy imagination, focusing data collection on unknown states from which the main policy can still return toward expert behavior. Experiments on locomotion, navigation and manipulation show consistent gains in interaction efficiency, imitation performance, and robustness, indicating that RECON directs real-environment interaction toward recovery regions around the expert distribution that are underexplored by prior methods, and thereby learns a world model better suited for imitation.

Problem

• Model-Based Imitation Learning
• Exploration
• World Model
• Recoverability
• Interaction Efficiency

Innovation

• Model-Based Imitation Learning
• Recoverability-Conditioned Exploration
• Epistemic Uncertainty
• Decoupled Policy and Explorer
• World Model