Easy-to-Use Shielding for Reinforcement Learning

2026-06-02

AI Summary

This work addresses the challenge of safe exploration in reinforcement learning by preventing harmful actions during policy training. It introduces tempestpy, an extension of the Tempest toolchain, which enables the first end-to-end integration of formal safety shields with the widely used Gymnasium reinforcement learning framework, substantially lowering the barrier to entry for practitioners. The proposed approach supports reliable shield synthesis in stochastic multi-agent settings and incorporates symbolic environment modeling. To facilitate reproducibility and broader adoption, the authors open-source both the tempestpy library and MiniGridSafe, a novel benchmark suite of safety-constrained environments. Empirical evaluations across multiple scenarios demonstrate the effectiveness of shielded reinforcement learning, enhancing both accessibility and experimental transparency in safe exploration research.

Abstract

Safe exploration is a key challenge in Reinforcement Learning (RL) that aims to prevent agents from making harmful decisions while exploring their environment. Safe exploration is a key challenge in Reinforcement Learning (RL) that aims to prevent agents from making harmful decisions while exploring their environment. Shielding is one such technique that assumes domain knowledge in the form of an environment model to decide upon action safety. Although well-established, shielding has seen limited adoption in RL due to the lack of accessible end-to-end infrastructure connecting formal shield synthesis with standard RL frameworks. Applying shielding typically requires expertise in formal methods and substantial engineering effort, keeping it outside the typical RL workflow. We address this by extending our shield synthesis tool Tempest into a practical backend for safe RL. Our core contribution is tempestpy, a Python library that integrates Tempest-based shield synthesis directly into the Gymnasium API, allowing shields to be synthesized and deployed within existing RL pipelines. This lowers the barrier to entry for shielding and turns formal safe-exploration methods into a usable component for RL practitioners. We also extend Tempest's algorithmic support to compute sound shields for stochastic multiplayer games, preserving formal safety guarantees. We demonstrate the resulting workflow end to end and evaluate shielded and unshielded RL across multiple environments. To facilitate modeling, we provide symbolic models for MiniGrid and introduce MiniGridSafe, a collection of playground environments designed to make shielding easily accessible and experimentally transparent. MiniGridSafe extends MiniGrid with safety-oriented scenarios featuring probabilistic transitions and additional agents, enabling the study of challenging safety aspects in a simple and intuitive setting.

Problem

• safe exploration
• reinforcement learning
• shielding
• formal methods
• stochastic multiplayer games

Innovation

• shielding
• safe reinforcement learning
• formal methods
• stochastic multiplayer games
• Gymnasium integration