FoSeRL: Formal Sequential Robustness Certification for Reinforcement Learning Policies

2026-10-03

AI Summary

This study addresses the challenge of certifying the performance degradation of reinforcement learning policies under sparse action attacks in stochastic environments. To this end, we propose the FoSeRL framework. By leveraging shared randomness and state augmentation, this method reduces complex trajectory-level certification to terminal safety verification. Furthermore, it employs time-dependent barrier conditions to achieve non-intrusive robustness certification without requiring smoothing or retraining. Experimental results demonstrate that FoSeRL attains certified budgets substantially exceeding those of policy smoothing methods across multiple environments, while effectively revealing robustness discrepancies among different policies.

Abstract

Even a few action perturbations can substantially degrade the performance of a deployed decision policy. Certifying the resulting return loss is challenging in stochastic environments, where returns vary even without an attack. We introduce FoSeRL, a framework for certifying deployed RL policies against precommitted, temporally sparse action attacks. The deployed policy is unchanged, with no smoothing or retraining. Certification requires a resettable simulator supporting shared randomness and independent one-step successor queries, but no analytical dynamics model. FoSeRL certifies that an attacked episode loses no more than a prescribed amount of return relative to the same episode unattacked, with at least a target probability and at a user-specified confidence level. Both runs share the initial state and randomness, so the measured loss reflects the attack, not the episode; carrying the running return gap as a state coordinate makes it the terminal value, reducing trajectory-level certification to terminal safety. Time-dependent barrier conditions on the augmented state bound the terminal failure probability: satisfied exactly, they certify every admissible precommitted attack; learned from sampled trajectories and verified on held-out data, they certify the same guarantee under a specified attack-episode setting. Across six stochastic continuous-control environments and three RL policy families (TD3, SAC, and PPO), FoSeRL certifies non-trivial cardinality--magnitude robustness frontiers, achieves substantially larger certified budgets than policy smoothing, and reveals marked robustness differences among policies with comparable nominal performance.

Problem

• Reinforcement Learning
• Robustness Certification
• Action Perturbations
• Stochastic Environments
• Sequential Robustness

Innovation

• Reinforcement Learning
• Robustness Certification
• Barrier Functions
• Action Perturbation
• Stochastic Environments