TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks

2025-09-30

AI Summary

Standard reinforcement learning struggles with complex, long-horizon control tasks specified in Signal Temporal Logic (STL) due to STL’s non-Markovian nature and sparse reward structure. To address this, we propose the first hierarchical optimization framework grounded in STL structural decomposition: STL formulas are parsed into temporally constrained subgoals and invariants, enabling a time-aware hierarchical policy architecture. Our method introduces a critic-guided temporal search mechanism coupled with Metropolis-Hastings sampling, phase-wise dense reward shaping, and differentiable feedback signals derived from STL robustness semantics. Evaluated across five benchmark environments, our approach achieves an average 31.6% improvement in task success rate over state-of-the-art methods, demonstrating superior generalization and stability—particularly in high-dimensional and long-horizon settings.

Technology Category

• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Learning control policies for complex, long-horizon tasks is a central challenge in robotics and autonomous systems. Signal Temporal Logic (STL) offers a powerful and expressive language for specifying such tasks, but its non-Markovian nature and inherent sparse reward make it difficult to be solved via standard Reinforcement Learning (RL) algorithms. Prior RL approaches focus only on limited STL fragments or use STL robustness scores as sparse terminal rewards. In this paper, we propose TGPO, Temporal Grounded Policy Optimization, to solve general STL tasks. TGPO decomposes STL into timed subgoals and invariant constraints and provides a hierarchical framework to tackle the problem. The high-level component of TGPO proposes concrete time allocations for these subgoals, and the low-level time-conditioned policy learns to achieve the sequenced subgoals using a dense, stage-wise reward signal. During inference, we sample various time allocations and select the most promising assignment for the policy network to rollout the solution trajectory. To foster efficient policy learning for complex STL with multiple subgoals, we leverage the learned critic to guide the high-level temporal search via Metropolis-Hastings sampling, focusing exploration on temporally feasible solutions. We conduct experiments on five environments, ranging from low-dimensional navigation to manipulation, drone, and quadrupedal locomotion. Under a wide range of STL tasks, TGPO significantly outperforms state-of-the-art baselines (especially for high-dimensional and long-horizon cases), with an average of 31.6% improvement in task success rate compared to the best baseline. The code will be available at https://github.com/mengyuest/TGPO

Problem

• Solving complex long-horizon tasks via STL specifications
• Overcoming non-Markovian nature and sparse reward challenges
• Decomposing STL into timed subgoals with hierarchical optimization

Innovation

• Decomposes STL into timed subgoals and constraints
• Uses hierarchical framework with time-conditioned policies
• Guides temporal search via learned critic sampling