🤖 AI Summary
This work investigates whether reinforcement learning agents truly acquire the underlying task states when achieving high rewards or merely exploit reward shortcuts. To disentangle reward acquisition from state representation, the authors construct controllable symbolic sequence tasks grounded in implicit deterministic finite automata (DFAs) with sparse terminal rewards. They introduce a white-box probing method that, for the first time, distinguishes between perceptual gaps and planning gaps, revealing that the linguistic structure of task families effectively predicts learning failure. Evaluated on 153 novel DFA-based tasks, this structural predictor achieves an accuracy of 0.86 in anticipating perceptual gaps. Experiments further demonstrate that while strong optimizers can partially recover latent states, they exhibit high variance, and critically, high reward does not necessarily imply genuine task understanding.
📝 Abstract
Does a reinforcement-learning agent that earns high reward represent its task's latent state, or only a reward-correlated shortcut? The question is usually unanswerable: the "true state" is undefined. We make it exactly answerable with a white-box instrument: express the task as a hidden deterministic finite automaton (DFA), let the agent observe a symbol stream and intermittently choose the next symbol under partial control, and grant one sparse terminal reward for acceptance. Knowing the automaton gives two things for free: the optimal return (so reward becomes an interpretable normalized score) and the exact latent state at every step (so we can probe the agent's representation without ever showing it). Reward success and latent-state learning thus become separately measured quantities whose coupling is governed by three controllable axes. Optimizer strength: under weak on-policy RL the agent earns reward with the state probe at chance for every architecture, tempting the conclusion that sparse RL cannot install latent state; a pre-registered control overturns it -- PPO+GAE recovers the state, but only partially and with high seed variance. Task structure: permutation (group-language) structure is a warning sign computable from the transition function before any training, and held out on 153 capacity-controlled fresh automata it flags perception gaps at precision 0.86 (89 of 103), in one direction only. Observation informativeness: a label-free auxiliary is vacuous when observations carry no state and recovers it in proportion to how much they reveal. The payoff is a distinction reward-only evaluation cannot make: a perception gap (latent state not linearly recoverable, though representable) versus a planning gap (state recoverable but unused). High reward is thus not evidence of task understanding; whether an agent recovers latent state is predictable in advance.