Double Horizon Model-Based Policy Optimization

2025-12-17

AI Summary

In model-based reinforcement learning (MBRL), rollout length entails a three-way trade-off: long rollouts exacerbate model bias and distributional shift, whereas short rollouts inflate policy gradient variance and impair value estimation. This work is the first to decouple the dual roles of rollouts—distribution alignment and gradient estimation—and proposes a two-timescale rollout mechanism: long rollouts mitigate state distribution shift, while short rollouts ensure stable, differentiable gradient computation. Our approach integrates hierarchical differentiable dynamics modeling, distribution-matching constrained optimization, and truncated backpropagation to overcome the multi-objective optimization bottleneck inherent in single-target alignment strategies. Evaluated on standard continuous-control benchmarks, the method significantly improves sample efficiency and training speed: convergence steps decrease by 32%, and total runtime reduces by 27%.

Technology Category

• Machine Learning: Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Model-based reinforcement learning (MBRL) reduces the cost of real-environment sampling by generating synthetic trajectories (called rollouts) from a learned dynamics model. However, choosing the length of the rollouts poses two dilemmas: (1) Longer rollouts better preserve on-policy training but amplify model bias, indicating the need for an intermediate horizon to mitigate distribution shift (i.e., the gap between on-policy and past off-policy samples). (2) Moreover, a longer model rollout may reduce value estimation bias but raise the variance of policy gradients due to backpropagation through multiple steps, implying another intermediate horizon for stable gradient estimates. However, these two optimal horizons may differ. To resolve this conflict, we propose Double Horizon Model-Based Policy Optimization (DHMBPO), which divides the rollout procedure into a long "distribution rollout" (DR) and a short "training rollout" (TR). The DR generates on-policy state samples for mitigating distribution shift. In contrast, the short TR leverages differentiable transitions to offer accurate value gradient estimation with stable gradient updates, thereby requiring fewer updates and reducing overall runtime. We demonstrate that the double-horizon approach effectively balances distribution shift, model bias, and gradient instability, and surpasses existing MBRL methods on continuous-control benchmarks in terms of both sample efficiency and runtime.

Problem

• Balancing distribution shift and model bias in MBRL rollouts
• Reducing gradient variance while maintaining accurate value estimation
• Optimizing rollout horizons for sample efficiency and runtime

Innovation

• Double horizon rollout for distribution and training
• Long distribution rollout mitigates on-policy shift
• Short training rollout ensures stable gradient updates