Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes

2026-07-29

AI Summary

In partially observable environments, agents must rely on a minimal Markov sufficient statistic to recover Markovian dynamics, yet this statistic is typically unknown. This work addresses a class of structured POMDPs—Holonomy-Cover decision processes—and introduces the “stable quotient,” the coarsest observation abstraction that preserves one-step rewards and successor relationships. It proves that the current observation together with its stable equivalence class constitutes an exact finite Markov state. Under conditions separating reachability and decision-making, the stable quotient achieves minimal memory usage in terms of symbolic representation. Coupled with a calibration-and-restart mechanism, it extends theoretical guarantees from finite MDPs to state recovery. Experiments demonstrate that perfect pairwise ordering accuracy can be achieved using memory from only three decision steps, matching the performance of a quotient oracle and significantly outperforming non-oracle baselines.

Abstract

An agent acting under partial observability must retain a recursively updateable statistic of history that restores the Markov property, but the smallest such statistic is generally unknown. We characterize this minimal Markov sufficient statistic for holonomy-cover decision processes, a structured POMDP class in which the visible dynamics are Markov and every realized visible transition applies a fixed permutation to a hidden mode. In particular, we construct the stable quotient, the coarsest observation-wise abstraction preserving one-step rewards and quotient successors, and prove that the pair of the current observation and stable class forms an exact finite Markov state. When the current class is correctly initialized, exact class tracking requires exactly the minimal memory symbols, in the sense that under reachability and pairwise decision separation at a maximizing observation, no arbitrary finite-memory controller can use fewer. Under resettable diagnostics, nearest-prototype class inference has exponentially decaying error, and a calibrate-then-restart reduction transfers finite-MDP guarantees to the recovered state. The results enable \emph{Holonomy Memory Reinforcement Learning}. It represents memory by the current stable class, updates it through ordered edge transports, identifies local class coordinates when diagnostics are available, and applies a standard finite-MDP RL backbone after synchronization. Experiments recover an exact compression from raw states to quotient states and achieve perfect paired-order accuracy with three decision-time memory states, matching the quotient oracle and outperforming the non-oracle baselines.

Problem

• partial observability
• Markov property
• sufficient statistic
• POMDP
• minimal memory

Innovation

• stable quotient
• minimal Markovization
• holonomy-cover decision processes
• finite-memory controller
• Holonomy Memory Reinforcement Learning