On the Design of Safe Continual RL Methods for Control of Nonlinear Systems

2025-02-21

AI Summary

This work addresses the fundamental challenge of jointly ensuring safety and enabling continual learning in nonlinear, nonstationary systems—such as those subject to unknown faults or abrupt constraint changes—where conventional safe reinforcement learning (Safe RL) and continual RL methods fail to coexist. We identify and analyze the intrinsic mechanism by which continual learning erodes safety constraints. To resolve this, we propose a safety-prioritized, elastic reward shaping framework that online integrates Elastic Weight Consolidation (EWC) with Constrained Policy Optimization (CPO), thereby achieving synergistic optimization of safety constraint satisfaction and task performance stability. Extensive evaluation on MuJoCo (HalfCheetah, Ant) under diverse nonstationary fault scenarios—including joint failures and sudden velocity constraint shifts—demonstrates that our method achieves over 92% safety constraint satisfaction, reduces performance degradation by 67%, and significantly mitigates catastrophic forgetting. To the best of our knowledge, this is the first approach to provably reconcile Safe RL and continual RL in dynamic nonlinear systems.

Technology Category

Machine Learning: Life-Long and Continual LearningNatural Language Processing: Safety and RobustnessConstraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

Responsible Web: Machine-in-the-loop, human agency and autonomySearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingSecurity and Privacy: Large-scale security measurements

Abstract

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs in the system, typical safe RL algorithms are unable to adapt while retaining past knowledge. Continual reinforcement learning algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system's safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, online elastic weight consolidation, is unable to satisfy safety constraints in non-linear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, non-linear, and non-stationary dynamical systems.

Problem

• Safe continual RL for non-linear systems
• Adapting to changing operating conditions
• Preventing catastrophic forgetting in safety-critical tasks

Innovation

• Safe continual RL methods
• Reward-shaping for safety
• Non-linear system adaptation