Beyond Policy Alignment: Closing the Planning-Learning Loop for Robot Control with Learned World Models

2026-09-30

AI Summary

This study addresses the insufficient closed-loop feedback between planning and learning in robotic control by proposing the PL-MPC framework. Building upon the TD-MPC architecture, this method enhances the synergistic optimization of world model planning and policy learning without altering the original architecture. Specifically, it introduces a hybrid multi-step temporal-difference target to improve critic supervision, designs a divergence-aware terminal value estimation, and employs return-weighted actor distillation. Experimental results demonstrate that PL-MPC significantly improves training efficiency on the HumanoidBench benchmark and achieves zero-shot sim-to-real transfer. Furthermore, it surpasses existing baselines in success rate on KUKA manipulator tasks, highlighting its effectiveness in bridging planning and learning for complex robotic control.

Abstract

Planning with learned world models combines online trajectory optimization with learned value and policy functions for high-dimensional control. Because the planner determines the experience used for learning, while the learned critic and actor in turn score and propose future plans, planning and learning form a closed feedback loop. TD-MPC is a prominent instance of this design. Recent policy-constrained variants strengthen one part of the loop by aligning the learned policy with planner behavior. We introduce PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation. Hybrid multi-step TD targets expose critic updates to more realized rewards before bootstrapping; disagreement-aware terminal estimates reduce the influence of uncertain critic values during MPPI planning; and return-weighted actor distillation emphasizes planner-executed actions from high-return episodes. The world-model architecture and MPPI optimizer are otherwise unchanged. On HumanoidBench, the largest gains occur on \texttt{balance-hard}, where Total Average Return (TAR) increases from $98\pm18$ to $387\pm255$, and \texttt{hurdle}, from $199\pm13$ to $466\pm200$; performance across the broader benchmark remains task dependent, and PL-MPC remains competitive on DMControl. Controlled ablations show different component interactions across the two tasks. We further demonstrate zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2 on the training object size and two unseen sizes. Code and data will be available at: https://pl-mpc-humanoid.github.io.

Problem

• World Models
• Robot Control
• Planning-Learning Loop
• Model Predictive Control
• Sim-to-Real Transfer

Innovation

• World Models
• Model Predictive Control
• Planning-Learning Loop
• Sim-to-Real Transfer
• Multi-step TD