Safe on Average, Unsafe in the Tail: When Is the Episodic-Cost Tail Controllable?

2026-10-07

AI Summary

This study addresses the limitation of safe reinforcement learning, where constraining only the expected cost overlooks tail risks and leads to uncontrolled worst-case cost violations. To mitigate this issue, this work introduces Conditional Value-at-Risk (CVaR) to quantify tail risks and proposes a CVaR-based tail-safety classification criterion. Leveraging the Safety-Gymnasium benchmark alongside various constrained algorithms, the authors systematically evaluate whether policies can bound worst-case trajectory costs within predefined budgets across navigation and locomotion tasks. The results successfully identify multiple policies that satisfy average safety constraints yet exhibit severe tail violations, thereby bridging the gap left by conventional evaluation paradigms that fail to reveal tail risks. Furthermore, this research quantitatively characterizes the capacity of different algorithms to control tail risks in safety-critical settings.

Abstract

Safe reinforcement learning seeks policies that maximize return while satisfying constraints on cumulative cost. Most methods impose these constraints on expected episodic cost. Consequently, standard evaluations report mean episodic cost without characterizing how cost is distributed across episodes. A policy that satisfies the mean-cost criterion may therefore remain unsafe in its worst episodes. Mean-cost reporting neither identifies this tail violation nor shows whether it can be brought within budget while preserving return. In this work, we measure the episodic-cost tail using $\mathrm{CVaR}_{0.1}$, the average cost of the worst $10\%$ of episodes. We classify a policy as tail-safe when $\mathrm{CVaR}_{0.1}$ is within the safety budget. This allows us first to identify policies that are safe on average but unsafe in the tail and then to study whether their tail violations can be controlled while preserving return. To identify tail-unsafe policies, we evaluate five standard algorithms on three Safety-Gymnasium navigation tasks. We then examine four constraint families on dense-hazard navigation and assess tail control across four navigation and four locomotion tasks.

Problem

• Safe Reinforcement Learning
• Episodic Cost Tail
• CVaR
• Tail Safety
• Constraint Satisfaction

Innovation

• Safe Reinforcement Learning
• Tail Risk
• CVaR
• Episodic Cost
• Constraint Satisfaction