Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learning

2026-09-27

AI Summary

This study addresses the challenge of modeling multimodal action distributions under hard state constraints in online safe reinforcement learning by proposing the SSM algorithm. This method leverages diffusion policies to represent multimodal distributions and, for the first time, extends Q-score matching to hard-constrained settings. By integrating Hamilton-Jacobi reachability theory, it designs a gated dual-branch scoring objective that dynamically switches between value optimization within feasible regions and recovery behaviors outside them, effectively balancing reward maximization with safety. Experimental results demonstrate that SSM achieves state-of-the-art performance on benchmark tasks such as UAV trajectory tracking, significantly reducing the false safety rate compared to existing baselines.

Abstract

Online safe reinforcement learning (RL) seeks policies that maximize reward while satisfying safety constraints. A popular line of research in safe RL relaxes safety to a soft expected-cost constraint and solves the resulting Constrained Markov Decision Process via primal-dual Lagrangian updates that only enforce safety on average. To address this limitation, hard, state-wise constraints are introduced and often imposed through Hamilton-Jacobi (HJ) reachability. Yet such constraints require solving different objectives in the feasible and infeasible regions: reward maximization in the former, recovery toward the feasible regions in the latter. The resulting target action distributions are inherently multimodal, and this structure poses a fundamental challenge for the Gaussian or deterministic actors used in existing HJ-based safe RL, which often collapse onto suboptimal modes. Diffusion policies provide the expressiveness needed to represent such distributions, and recent work on Q-score matching offers a route to training them for online RL by score regression -- but has been applied only to reward maximization. We propose Safe Score Matching (SSM), an off-policy actor-critic method that adapts Q-score matching to hard-constrained safe RL by gating a two-branch score target with HJ reachability: inside the feasible set, the denoising process degenerates to Q-score matching on actions classified as viable by the HJ critic; outside, a recovery branch biases denoising toward regions with lower worst-case violation. On quadrotor and fixed-wing trajectory-tracking and stabilize-and-avoid benchmarks, SSM attains the best or near-best task performance with low false-safe rates, whereas the primal-dual baseline admits more unsafe behavior and reachability-based baselines tend to be more conservative; on Safety-Gymnasium velocity tasks, SSM attains the lowest cost with competitive reward.

Problem

• safe reinforcement learning
• hard constraints
• multimodal action distribution
• diffusion policy
• Hamilton-Jacobi reachability

Innovation

• Safe Score Matching
• Diffusion Policies
• Hamilton-Jacobi Reachability
• Q-score Matching
• Safe Reinforcement Learning