Robust Shielding for Safe Reinforcement Learning

2026-05-29

AI Summary

This work addresses the challenge of ensuring safety in reinforcement learning under unknown environmental dynamics, where existing shielding methods—reliant on known safe transition models—often fail. The paper proposes the first safety shielding framework tailored for robust Markov decision processes (RMDPs), which guarantees satisfaction of linear temporal logic (LTL) specifications with high confidence under worst-case transition dynamics. By integrating formal verification with probably approximately correct (PAC) sampling, the approach extends shielding mechanisms to the RMDP setting while simultaneously ensuring both safety and near-optimality of the learned policy. Empirical results demonstrate that the proposed shield effectively enforces safety in previously unseen environments and progressively recovers near-optimal expected returns as the number of collected samples increases.

Abstract

Shielding is an effective approach to formally guarantee the safety of reinforcement learning agents in Markov decision processes (MDPs). However, existing shielding techniques typically assume knowledge of the safety-relevant transition dynamics - a requirement that is seldom met in practice. To address this limitation, we introduce a novel shielding framework for robust MDPs (RMDPs), i.e., MDPs with sets of transition probabilities. We define safety as the satisfaction of a linear temporal logic (LTL) formula with a certain threshold probability under the worst-case transition probabilities of the RMDP. We prove that our shielding framework is both sound and optimal for the RMDP: every policy admissible by the shield is safe, and conversely, every safe RMDP policy is admissible by the shield. We combine our approach with existing sampling methods for learning transition probabilities of MDPs with probably approximately correct (PAC) guarantees. This combination enables the construction of shields for MDPs that, with high confidence, guarantee safety while remaining minimally restrictive. Our experiments show that our shields for learned RMDPs guarantee safety in unknown MDPs while recovering strong expected return as the number of samples increases.

Problem

• shielding
• safe reinforcement learning
• robust MDPs
• safety guarantees
• unknown transition dynamics

Innovation

• Robust MDP
• Shielding
• Linear Temporal Logic
• Safe Reinforcement Learning
• PAC Learning