🤖 AI Summary
This work addresses the performance degradation of real-world reinforcement learning systems under the common “train-then-deploy” paradigm, which fails to adapt to dynamic environmental changes. To overcome this limitation, the paper introduces a novel “deploy-and-continuously-learn” paradigm that treats deployed agents as lifelong reinforcement learning systems. It identifies four key sources of non-stationarity encountered post-deployment and integrates online adaptation, explicit non-stationarity modeling, and lifelong learning mechanisms to maintain robust performance. Through analysis of real-world deployment scenarios, the study demonstrates the advantages of this paradigm in sustaining long-term effectiveness. Furthermore, it proposes new evaluation metrics tailored to continuous learning settings, aiming to shift the research community’s focus from static models toward lifelong learning agents capable of enduring environmental dynamics.
📝 Abstract
Reinforcement Learning (RL) has received increasing attention and adoption in real-world use cases. Most of these systems follow a train-then-fix paradigm, where trained agents do not learn while interacting with the world until performance degrades and retraining becomes necessary. In this position paper, we argue that deploying an agent that is incapable of optimality, but receives an evaluative reward signal, is inherently a continual RL problem. We identify four sources of non-stationarity after deployment that necessitate never-ending learning, and highlight why the best deployed agents never stop adapting. We analyze successful examples of continual RL in the real world, and present the community with the advantages and measures to move away from the current train-then-fix paradigm.