Innovation
Methods, ideas, or system contributions that make the work stand out.
Self-Confirming Superposition Traps
Representation Learning
Replay Condition
Feature Interference
Protected Fitting
Self-Confirming Superposition Traps in Reinforcement Learning
🤖 AI Summary
This study addresses the self-confirming superposition trap in reinforcement learning, wherein policy-selected data distorts learned representations and locks agents into suboptimal policies. We provide the first formal definition of this phenomenon, theoretically demonstrating that even globally optimal fitting can converge to local optima. To mitigate this, we propose an intervention mechanism combining experience replay with state protection, which disrupts feedback loops by preserving access to neglected states. Experiments integrating our method with PPO and DreamerV3 across MiniGrid, DMControl, and Crafter environments demonstrate effective suppression of measurement interference and significant improvements in episodic returns. Notably, performance gains in cumulative training scores persist even when the encoder is frozen, confirming the robustness of our approach against representation collapse induced by policy-dependent data collection.
📝 Abstract
Reinforcement learning (RL) trains representations on data selected by the agent's policy, which then uses the resulting returns to guide its next choices. We show that this loop can sustain a lower-return policy even when representation fitting is globally optimal on those data. In a self-confirming superposition trap, every optimal code assigns overlapping directions to features that rarely occur together under the current policy. An alternative action brings them together, causing interference that lowers its return and reinforces avoidance, although refitting to that action would yield more return at the same capacity. We characterize the dimensions admitting a trap in a tied two-step model and show separately that equal feature frequencies, continued visitation, and independent controller learning need not prevent it. Because fitting weights errors by visitation, an avoided action can lose its return advantage at little cost to the objective. In a finite-action model, we bound this distortion and derive a replay condition: sufficient training weight on the best separately adapted action preserves its ranking despite residual error. Neural PPO experiments show how the feedback develops during learning: agents initialized toward different actions develop different interference patterns, opposite mean return rankings, and different final policies at the same capacity. We therefore test whether retaining access to neglected states can improve control. Keeping these states in training reduces measured interference and improves sequential return, with gains even when the encoder is frozen. Related interventions on state access, replay weights, and feature overlap improve control on MiniGrid and DMControl. For agents that learn through a world model, protected fitting improves DreamerV3--Crafter's cumulative training scores at unchanged capacity.
🔎 Similar Papers
No similar papers found.
