Good-for-MDP State Reduction for Stochastic LTL Planning

2025-11-12

AI Summary

To address the scalability bottleneck in MDP policy synthesis under LTL objectives—caused by state-space explosion during generalized Rabin automaton (GFM) construction—this paper proposes a novel state-space reduction framework. Methodologically, it introduces (1) a game-theoretically optimal “good-for-games minimization” technique, integrating formal translation with specialized GFM construction, and (2) for the key LTL fragment $mathsf{G}mathsf{F}varphi$, a direct GFM construction algorithm achieving single-exponential time complexity, breaking the classical double-exponential barrier. Experimental evaluation on standard benchmarks demonstrates that the approach reduces automaton size by one to two orders of magnitude, significantly improving policy synthesis efficiency and overall scalability. These advances provide a practical pathway for large-scale LTL-constrained MDP planning.

Technology Category

• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We study stochastic planning problems in Markov Decision Processes (MDPs) with goals specified in Linear Temporal Logic (LTL). The state-of-the-art approach transforms LTL formulas into good-for-MDP (GFM) automata, which feature a restricted form of nondeterminism. These automata are then composed with the MDP, allowing the agent to resolve the nondeterminism during policy synthesis. A major factor affecting the scalability of this approach is the size of the generated automata. In this paper, we propose a novel GFM state-space reduction technique that significantly reduces the number of automata states. Our method employs a sophisticated chain of transformations, leveraging recent advances in good-for-games minimisation developed for adversarial settings. In addition to our theoretical contributions, we present empirical results demonstrating the practical effectiveness of our state-reduction technique. Furthermore, we introduce a direct construction method for formulas of the form $mathsf{G}mathsf{F}varphi$, where $varphi$ is a co-safety formula. This construction is provably single-exponential in the worst case, in contrast to the general doubly-exponential complexity. Our experiments confirm the scalability advantages of this specialised construction.

Problem

• Reducing automata state size for scalable LTL planning in MDPs
• Developing efficient state-space reduction for good-for-MDP automata
• Creating specialized construction for GFφ formulas with better complexity

Innovation

• GFM state-space reduction technique for automata
• Chain of transformations using good-for-games minimisation
• Direct single-exponential construction for GFφ formulas