Embedding Semantic Risk into Distance Fields and CBFs for Online Monocular Safe Control

2026-05-31

AI Summary

This work proposes an end-to-end monocular visual safety control framework that explicitly models category-dependent risk by integrating geometric and semantic information prior to constructing the Euclidean Signed Distance Field (ESDF)—a departure from existing approaches that either employ a uniform safety margin or incorporate semantics only in post-control stages. By leveraging a foundation-model-based SLAM frontend for dense 3D reconstruction and per-frame semantic segmentation, the system generates a semantic-aware ESDF, which provides both distance values and gradients for Control Barrier Functions (CBFs). This enables risk-informed safety constraints where high-risk objects exert larger spatial influence in the safety field. Experiments demonstrate that the method operates in real time at 10–20 Hz on both simulated and real-world platforms, significantly enhancing semantic-aware safety in both teleoperation and autonomous navigation scenarios.

Abstract

We propose an online monocular perception-to-control framework that embeds semantic risk into the distance field used by Control Barrier Function (CBF)-based safe navigation and teleoperation. Many perception-based safety filters assign the same distance-based safety margin to all mapped obstacles or use semantics only as a downstream controller adjustment, rather than encoding semantic risk in the spatial representation. Our framework instead reasons online about obstacle geometry and class-dependent risk by embedding semantic information directly into the Euclidean Signed Distance Field (ESDF). This design encodes semantic risk before control optimization, so high-risk objects exert a larger spatial influence in the safety field while retaining efficient ESDF queries at runtime. Specifically, a foundation-model-based SLAM front end reconstructs dense 3-D geometry from monocular RGB video, while per-frame semantic segmentation provides pixel-level class labels that are fused into the reconstructed geometry. The resulting geometric-semantic representation is then converted into an ESDF, where semantic labels identify safety-relevant regions and impose class-dependent inflation before field computation. The semantic-aware ESDF provides the local distance values and spatial derivatives required by the CBF controller, while class-dependent gains further regulate the controller response. Extensive simulation and hardware experiments demonstrate online operation at 10--20 Hz and semantic-aware safe behavior in both teleoperation and autonomous navigation.

Problem

• semantic risk
• distance fields
• Control Barrier Functions
• monocular perception
• safe navigation

Innovation

• Semantic Risk
• Control Barrier Functions
• Euclidean Signed Distance Field
• Monocular SLAM
• Semantic Segmentation