Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning

2026-07-27

AI Summary

In continual reinforcement learning, neural networks often suffer from policy collapse due to neuron dormancy and degraded representational capacity. This work proposes the Calibrated Partial Reset (CPR) optimizer, which introduces, for the first time, a utility-weighted partial reinitialization mechanism. CPR dynamically modulates the strength with which neurons are pulled back toward their initial states based on their utility, thereby preserving model plasticity while avoiding the performance degradation and training instability associated with full resets. The method effectively maintains stable gradient flow in non-stationary environments. Empirical results demonstrate that CPR is the only approach capable of preventing policy collapse within 400 million steps on SlipperyAnt and significantly outperforms existing decay and reset strategies on the Continual MetaWorld and Continual MinAtar benchmarks.

Abstract

Neural networks are hindered by accumulating dormant neurons and loss of expressivity throughout training, particularly in non-stationary data settings, such as continual supervised and reinforcement learning. Recently, neuron resets have been used to maintain gradient flow and restore plasticity. However, full unit reinitialization often sacrifices peak performance and can destabilize training, leading to policy collapse. To preserve plasticity without destabilizing training, we propose Calibrated Partial Resets (CPR), an optimizer that periodically pulls low-utility neurons toward their initialization, with pull strength scaled by each neuron's utility. Unlike binary reset methods, partial resets avoid brittleness; unlike uniform decay, calibrated utility-scaling concentrates adjustment on the units that need it most. Among compared methods, only CPR avoids policy collapse over 400M training steps in SlipperyAnt, and it outperforms prior decay and reset-based methods on Continual MetaWorld and Continual MinAtar benchmarks. Ablations reveal a tunable trade-off between plasticity and peak performance, highlighting utility-scaled reinitialization as a promising direction for continual learning.

Problem

• continual reinforcement learning
• policy collapse
• dormant neurons
• loss of expressivity
• neural plasticity

Innovation

• Calibrated Partial Resets
• continual reinforcement learning
• neuron reset
• plasticity
• policy collapse