Vision--Language Signals in Constrained RL: Safety Gains Without Anticipation

2026-09-28

AI Summary

This study addresses the challenges of sparse collision costs, the absence of early warnings in reinforcement learning, and the unclear safety mechanisms of vision-language models (VLMs) by proposing the VLM-Safe-RL framework. Built upon the PPO-Lagrangian algorithm, this method integrates semantic signals from a frozen CLIP model into policy optimization via reward shaping and Lagrange multiplier updates. Our analysis reveals that VLM signals do not operate by predicting collisions or driving multiplier dynamics; instead, they facilitate conditional catastrophe avoidance through enhanced semantic feedback. Evaluated on the MetaDrive Hard benchmark, the proposed approach significantly reduces the catastrophe rate from 31.6% to 19.4%. These results validate the effectiveness of VLM-guided safe reinforcement learning even in the absence of anticipatory cues.

Abstract

Safe reinforcement learning seeks policies that maximise task performance while satisfying safety constraints. In driving benchmarks, however, collision costs typically appear only at the time of collision, providing no advance warning of an approaching hazard. Frozen vision--language models can provide dense semantic feedback, yet it remains unclear whether their scores anticipate collisions and which component drives an observed safety improvement. Episodic cost can also favour policies that make little task progress. To address these gaps, we propose VLM-Safe-RL, a framework that integrates frozen CLIP signals into PPO-Lagrangian through reward shaping and an augmented multiplier update. On MetaDrive Hard, which combines the densest traffic with the largest map, the catastrophe rate falls from 31.6\% to 19.4\%. FormulaOne-L2 analysis finds no evidence that the CLIP signals anticipate collisions and shows that the VLM term has a negligible effect on the Lagrange multiplier. These findings show a conditional reduction in observed catastrophe rate without evidence of collision anticipation.

Problem

• Safe Reinforcement Learning
• Vision-Language Models
• Collision Anticipation
• Safety Constraints

Innovation

• Safe Reinforcement Learning
• Vision-Language Models
• PPO-Lagrangian
• Reward Shaping
• CLIP