RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment

2026-10-06

AI Summary

This study addresses the safety risks faced by embodied agents operating under real-time constraints in physical environments, where inference latency can transform otherwise safe actions into hazardous ones. For the first time, this work incorporates inference latency into a safety evaluation framework by integrating vision-language models, reinforcement learning, and high-fidelity physics simulation to construct a simulated urban benchmark for systematically assessing real-time safety performance in dynamic settings. Our findings reveal that static evaluations obscure critical real-time safety vulnerabilities: real-time execution increases collision rates by 12.3-fold. Furthermore, we demonstrate that training via offline reinforcement learning significantly mitigates accident risks while preserving task completion rates.

Abstract

Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments. As agents move into the physical world, embodied safety becomes increasingly important: failures can cause human injury and costly hardware damage. Beyond selecting safe actions, embodied agents must also operate under real-time constraints: the physical world does not pause while an agent reasons. As pedestrians move and vehicles approach during inference, an action that appears safe at observation time may become unsafe before execution. Real-time embodied safety therefore depends on both decision quality and decision latency. We introduce RT-SAFE, a simulated urban benchmark for evaluating embodied-agent safety under real-time constraints. RT-SAFE combines navigation tasks with moving actors, environmental hazards, and traffic rules, while allowing the world to evolve throughout inference and action execution. Across eight VLMs, agents achieve high task completion yet almost never complete safely: in the hardest setting, only 0.7% of episodes finish without a safety event. More strikingly, matched static and real-time evaluations yield task completion rates of 91.3% and 94.1%, respectively, while real-time execution increases collisions by $12.3\times$. These results reveal that standard task success can mask substantial safety failures, and that decision latency itself can become a source of physical risk. Finally, we show that RT-SAFE can support offline RL training and substantially reduce collision rates while achieving strong task completion.

Problem

• Embodied Agent Safety
• Real-time Constraints
• Decision Latency
• Benchmarking
• Physical Risk

Innovation

• Embodied Agent Safety
• Real-time Benchmark
• Decision Latency
• Offline Reinforcement Learning
• Dynamic Simulation