RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences

2026-07-15

AI Summary

This work addresses the challenge in offline reinforcement learning where world models are prone to over-exploitation in data-scarce regions, leading to distorted dynamics predictions. The authors propose RENEW, a novel approach that, for the first time, directly leverages human preference feedback over imagined trajectories to supervise world model learning. By formulating a preference loss based on the Bradley-Terry model and integrating epistemic uncertainty to guide active fine-tuning, RENEW selectively refines the model in regions most susceptible to erroneous exploitation. This method significantly improves sample efficiency, effectively mitigates catastrophic forgetting, and substantially reduces the reliance on preference data in both Jumanji and classic control benchmarks. Consequently, RENEW curbs excessive model exploitation and enhances the practicality of offline model-based reinforcement learning.

Abstract

World models are widely used in offline reinforcement learning (RL) to improve sample efficiency and generate experience beyond a fixed dataset. However, they are vulnerable to model exploitation where data coverage is thin. Prior work addresses this either by collecting more expert demonstrations, which is often expensive, unsafe, or unavailable, or by conservative algorithms that avoid uncertain regions, which limits generalization. We propose instead to repair exploitation directly using human preferences over imagined rollouts, leveraging the strong intuitive physics that allows humans to easily spot egregious dynamics hallucinations. We formalize this as Dynamics Learning from Human Feedback (DLHF), a Bradley-Terry preference loss over trajectory log-likelihoods under a learned dynamics model. Unfortunately, naive DLHF is sample inefficient, so we introduce RENEW, which uses epistemic uncertainty to focus finetuning where the model is most exploitable. We evaluate on several Jumanji and classic control environments and find that while naive DLHF requires an outsize preference budget, RENEW makes the framework practical by improving sample efficiency, limiting catastrophic forgetting, and reducing exploitation in pretrained world models. Taken together, our results provide initial evidence that preferences can supervise world model dynamics directly, offering a new approach to addressing exploitation in offline model-based RL.

Problem

• world models
• model exploitation
• offline reinforcement learning
• human preferences
• sample efficiency

Innovation

• world models
• human feedback
• model exploitation
• offline reinforcement learning
• epistemic uncertainty