Continual Reinforcement Learning by Planning with Online World Models

2025-07-12

AI Summary

To address catastrophic forgetting in continual reinforcement learning (CRL), this paper proposes a planning-based continual learning framework grounded in an online world model. The core method introduces a Follow-The-Leader (FTL) online shallow modeling approach to learn task-invariant world dynamics, yielding a theoretical regret bound of $O(sqrt{K^2 D log T})$ without relying on experience replay or parameter regularization—thereby inherently mitigating forgetting. By integrating online learning with model-predictive control, the framework enables efficient action planning via incremental model updates, resulting in the FTL Online Agent (OA). Evaluated on the Continual Bench benchmark, OA acquires new skills sequentially across multi-task streams with zero forgetting, consistently outperforming deep-model-based and state-of-the-art continual learning baselines.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Continual reinforcement learning (CRL) refers to a naturalistic setting where an agent needs to endlessly evolve, by trial and error, to solve multiple tasks that are presented sequentially. One of the largest obstacles to CRL is that the agent may forget how to solve previous tasks when learning a new task, known as catastrophic forgetting. In this paper, we propose to address this challenge by planning with online world models. Specifically, we learn a Follow-The-Leader shallow model online to capture the world dynamics, in which we plan using model predictive control to solve a set of tasks specified by any reward functions. The online world model is immune to forgetting by construction with a proven regret bound of $mathcal{O}(sqrt{K^2Dlog(T)})$ under mild assumptions. The planner searches actions solely based on the latest online model, thus forming a FTL Online Agent (OA) that updates incrementally. To assess OA, we further design Continual Bench, a dedicated environment for CRL, and compare with several strong baselines under the same model-planning algorithmic framework. The empirical results show that OA learns continuously to solve new tasks while not forgetting old skills, outperforming agents built on deep world models with various continual learning techniques.

Problem

• Address catastrophic forgetting in continual reinforcement learning
• Plan tasks using online world models with model predictive control
• Develop an incremental FTL Online Agent to solve sequential tasks

Innovation

• Planning with online world models
• Follow-The-Leader shallow model
• Model predictive control for tasks