Safe-RULE: Safe Reinforcement UnLEarning

2026-06-08

AI Summary

This work addresses the vulnerability of offline safe reinforcement learning to data poisoning attacks, which can cause learned policies to violate safety constraints. To counter this issue, the paper introduces, for the first time, a safe reinforcement unlearning paradigm that effectively mitigates the impact of poisoned data without requiring retraining or access to the original environment. By integrating offline reinforcement learning, explicit safety constraint modeling, and an unlearning mechanism, the proposed approach significantly enhances policy robustness and safety under adversarial attacks across multiple benchmark tasks, while preserving strong task performance.

Abstract

Offline safe reinforcement learning (Safe RL) enables policy learning without online interactions, making it suitable for safety-critical systems such as robotics systems. However, its reliance on static datasets exposes offline Safe RL to data poisoning attacks, where adversaries inject malicious samples that compromise safety and induce unsafe policy behavior. In this work, we propose a new learning paradigm, named safe reinforcement unlearning (Safe-RULE), used as a defense framework to remove the influence of poisoned data without retraining from scratch or requiring access to the original training environment. We further extend reinforcement unlearning to offline Safe RL by explicitly accounting for both task performance and safety constraints during the unlearning process. Experiments across benchmark Safe RL tasks demonstrate that our approach effectively enhances safety performance against data poisoning attacks.

Problem

• offline safe reinforcement learning
• data poisoning attacks
• safety constraints
• malicious samples
• unsafe policy behavior

Innovation

• safe reinforcement unlearning
• data poisoning defense
• offline safe RL
• policy safety
• unlearning without retraining