Property-driven Causal Abstractions for Markov Decision Processes

2026-07-29

AI Summary

This work addresses the scalability bottleneck in Markov decision processes (MDPs) caused by the exponential explosion of state spaces by proposing a property-oriented causal abstraction framework. It introduces causal reasoning into MDP abstraction for the first time, leveraging causal relationships among predicates over state variables to cluster states that either satisfy or violate a given target property, thereby constructing compact yet property-preserving abstract models. The framework accommodates various model types—including MDPs, interval MDPs, and stochastic games—and generates small abstract models on standard benchmarks that enable efficient computation of near-optimal policies. Empirical evaluation demonstrates strong generalization across large-scale MDPs, effectively balancing model fidelity with compression efficiency.

Abstract

Markov Decision Processes (MDPs) are widely used as decision-making models, commonly specified over factored state spaces through state variables and their valuations. The exponential blowup in the number of states renders many reasoning tasks in MDPs challenging. Abstractions are promising techniques to reduce MDPs and thus mitigate scalability issues. In this work, we introduce a notion of causality on factored MDPs and a novel property-driven causal abstraction technique that retains many characteristics of the original MDP model. For this, we rely on causal relations over state variable predicates and identify those states that share the same reasons for fulfilling or violating a given abstraction property. We theoretically and empirically compare various causal MDP abstractions using different model types such as MDPs, interval MDPs, or stochastic games. Our evaluation demonstrates the potential of our approach: For several standard benchmarks, we obtain small abstractions that allow us to compute near-optimal policies for the original MDP. Furthermore, our causal abstractions often generalize to related large-scale MDP models.

Problem

• Markov Decision Processes
• state-space explosion
• abstraction
• causality
• scalability

Innovation

• causal abstraction
• property-driven
• factored MDPs
• state space reduction
• near-optimal policy