Planning as Descent: Goal-Conditioned Latent Trajectory Synthesis in Learned Energy Landscapes

2025-12-19

AI Summary

This work addresses the challenges of trajectory feasibility and goal consistency verification in offline goal-conditioned reinforcement learning. We propose an energy-optimization framework that requires neither a policy network nor an explicit planner. Our method models trajectory planning as gradient descent on a goal-conditioned energy function in latent space; incorporates counterfactual goal relabeling for self-supervised energy shaping; and employs multi-step parallel refinement to enhance robustness. Crucially, we introduce the first differentiable energy function—replacing conventional policies or planners—to unify training and inference. Evaluated on the OGBench cube manipulation task, our approach achieves 95% success rate using only narrow-band expert demonstrations, substantially outperforming the state-of-the-art (68%). Moreover, it exhibits robustness to noisy data: training with such data further improves both success rate and planning efficiency.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments

Abstract

We present Planning as Descent (PaD), a framework for offline goal-conditioned reinforcement learning that grounds trajectory synthesis in verification. Instead of learning a policy or explicit planner, PaD learns a goal-conditioned energy function over entire latent trajectories, assigning low energy to feasible, goal-consistent futures. Planning is realized as gradient-based refinement in this energy landscape, using identical computation during training and inference to reduce train-test mismatch common in decoupled modeling pipelines. PaD is trained via self-supervised hindsight goal relabeling, shaping the energy landscape around the planning dynamics. At inference, multiple trajectory candidates are refined under different temporal hypotheses, and low-energy plans balancing feasibility and efficiency are selected. We evaluate PaD on OGBench cube manipulation tasks. When trained on narrow expert demonstrations, PaD achieves state-of-the-art 95% success, strongly outperforming prior methods that peak at 68%. Remarkably, training on noisy, suboptimal data further improves success and plan efficiency, highlighting the benefits of verification-driven planning. Our results suggest learning to evaluate and refine trajectories provides a robust alternative to direct policy learning for offline, reward-free planning.

Problem

• Develops offline goal-conditioned reinforcement learning via energy-based trajectory synthesis
• Uses gradient-based refinement in learned energy landscapes for planning
• Addresses train-test mismatch by integrating verification into trajectory evaluation

Innovation

• Learning goal-conditioned energy functions for latent trajectories
• Planning as gradient-based refinement in learned energy landscapes
• Self-supervised hindsight goal relabeling for training energy functions