Exact Model-Free Policy Iteration for Co-safe LTL Planning

2026-08-05

AI Summary

This work addresses the challenge in model-free reinforcement learning of satisfying syntactically co-safe linear temporal logic (sc-LTL) specifications, where non-contractive Bellman operators and non-unique solutions hinder policy convergence. To overcome this, the authors propose a two-stage approach: first, the sc-LTL specification is transformed into a maximum reachability objective, and a discounting surrogate is introduced to identify a clamping set that resolves solution ambiguity; second, undiscounted policy evaluation is performed followed by greedy policy improvement. This method uniquely integrates discounting surrogates with undiscounted policy iteration, theoretically guaranteeing almost-sure convergence of policy evaluation and finite-step termination of policy iteration at an optimal policy. Empirical validation in stochastic grid-world environments demonstrates the efficacy of the proposed framework.

Abstract

This work studies model-free reinforcement learning for co-safe linear temporal logic (sc-LTL) objectives in finite Markov decision processes, which can be reduced to maximal reachability objectives via the standard product construction. For this problem, direct sample-based bootstrap methods (e.g., TD or Q-learning) may fail to converge to optimal policies due to the noncontractive nature and nonuniqueness of solutions to the Bellman equation. We develop a new two-step model-free reinforcement learning method that first uses a discounted surrogate to identify a clamp set that resolves this nonuniqueness, and then applies undiscounted policy evaluation and greedy policy improvement with guarantees of finding an optimal solution. We prove almost-sure convergence of the policy evaluation step and finite termination of the policy iteration algorithm at an optimal policy. These theoretical results are validated through numerical experiments on a stochastic grid world.

Problem

• co-safe LTL
• model-free reinforcement learning
• Markov decision processes
• Bellman equation
• policy iteration

Innovation

• model-free reinforcement learning
• co-safe LTL
• policy iteration
• clamp set
• noncontractive Bellman equation