Demonstration-Guided Continual Reinforcement Learning in Dynamic Environments

2025-12-21

AI Summary

To address the challenge in continual reinforcement learning (CRL) of balancing knowledge stability and policy plasticity in dynamic environments, this paper proposes a novel CRL framework leveraging an external self-evolving demonstration library. Methodologically, it explicitly encodes prior knowledge as executable demonstrations and organizes them into a retrievable, autonomously updated external memory bank to directly guide agent exploration. Additionally, it introduces a curriculum-based progressive strategy that enables smooth transition from demonstration-guided behavior to autonomous policy optimization. The framework is seamlessly integrated with mainstream algorithms such as PPO and SAC. Empirical evaluation on 2D navigation and MuJoCo locomotion benchmarks demonstrates substantial improvements: +21.4% average performance gain, enhanced cross-task transferability, superior anti-catastrophic forgetting capability, and 37% higher training efficiency.

Technology Category

Machine Learning: Life-Long and Continual LearningMultiagent Systems: Multiagent LearningIntelligent Robots: Learning & Optimization for ROB

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsResponsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement learning (RL) excels in various applications but struggles in dynamic environments where the underlying Markov decision process evolves. Continual reinforcement learning (CRL) enables RL agents to continually learn and adapt to new tasks, but balancing stability (preserving prior knowledge) and plasticity (acquiring new knowledge) remains challenging. Existing methods primarily address the stability-plasticity dilemma through mechanisms where past knowledge influences optimization but rarely affects the agent's behavior directly, which may hinder effective knowledge reuse and efficient learning. In contrast, we propose demonstration-guided continual reinforcement learning (DGCRL), which stores prior knowledge in an external, self-evolving demonstration repository that directly guides RL exploration and adaptation. For each task, the agent dynamically selects the most relevant demonstration and follows a curriculum-based strategy to accelerate learning, gradually shifting from demonstration-guided exploration to fully self-exploration. Extensive experiments on 2D navigation and MuJoCo locomotion tasks demonstrate its superior average performance, enhanced knowledge transfer, mitigation of forgetting, and training efficiency. The additional sensitivity analysis and ablation study further validate its effectiveness.

Problem

• Addresses continual RL in dynamic environments
• Balances stability and plasticity via demonstrations
• Enhances knowledge transfer and reduces forgetting

Innovation

• External demonstration repository for knowledge storage
• Dynamic selection of relevant demonstrations for guidance
• Curriculum strategy from guided to self-exploration