Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning

2026-04-16

AI Summary

In continual reinforcement learning, maintaining a single policy often fails to balance knowledge retention with rapid adaptation, leading to loss of policy plasticity. To address this limitation, this work proposes the TeLAPA framework, which abandons the single-policy paradigm in favor of a diverse archive of task-specific policies. These policies are organized within a shared latent space that preserves skill alignment through neighborhood relationships. By integrating quality-diversity optimization, latent-space alignment, and an archival mechanism, TeLAPA enables effective organization, comparison, and reuse of policies in the MiniGrid continual learning environment. Experimental results demonstrate that TeLAPA learns a greater number of tasks in sequential settings, recovers more quickly after perturbations, and achieves significantly superior overall performance compared to methods relying on a single representative policy.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Continual reinforcement learning must balance retention with adaptation, yet many methods still rely on \emph{single-model preservation}, committing to one evolving policy as the main reusable solution across tasks. Even when a previously successful policy is retained, it may no longer provide a reliable starting point for rapid adaptation after interference, reflecting a form of \emph{loss of plasticity} that single-policy preservation cannot address. Inspired by quality-diversity methods, we introduce \textsc{TeLAPA} (Transfer-Enabled Latent-Aligned Policy Archives), a continual RL framework that organizes behaviorally diverse policy neighborhoods into per-task archives and maintains a shared latent space so that archived policies remain comparable and reusable under non-stationary drift. This perspective shifts continual RL from retaining isolated solutions to maintaining \emph{skill-aligned neighborhoods} with competent and behaviorally related policies that support future relearning. In our MiniGrid CL setting, \textsc{TeLAPA} learns more tasks successfully, recovers competence faster on revisited tasks after interference, and retains higher performance across a sequence of tasks. Our analyses show that source-optimal policies are often not transfer-optimal, even within a local competent neighborhood, and that effective reuse depends on retaining and selecting among multiple nearby alternatives rather than collapsing them to one representative. Together, these results reframe continual RL around reusable and competent policy neighborhoods, providing a route beyond single-model preservation toward more plastic lifelong agents.

Problem

• continual reinforcement learning
• plasticity loss
• single-model preservation
• policy reuse
• non-stationary drift

Innovation

• Continual Reinforcement Learning
• Policy Archives
• Behavioral Diversity
• Latent Alignment
• Plasticity Preservation