RADIO-ViPE: Online Tightly Coupled Multi-Modal Fusion for Open-Vocabulary Semantic SLAM in Dynamic Environments

2026-04-28

AI Summary

This work proposes the first online, open-vocabulary semantic SLAM system that operates without depth sensors, camera calibration, or initial pose estimates, enabling association of arbitrary natural language queries with 3D scene regions even in dynamic environments. By tightly coupling vision–language multimodal embeddings—leveraging foundation models such as RADIO—with geometric constraints, the method integrates adaptive robust kernels into factor graph optimization to handle moving objects and scene changes. Relying solely on raw monocular RGB video streams, the system achieves high-fidelity semantic–geometric consistency in dynamic scenes, attaining state-of-the-art performance on the TUM-RGBD dynamic benchmark. Remarkably, it matches the accuracy of offline approaches that assume calibrated setups and static scenes, thereby significantly expanding the applicability of semantic SLAM to real-world robotics and unconstrained video settings.

Abstract

We present RADIO-ViPE (Reduce All Domains Into One -- Video Pose Engine), an online semantic SLAM system that enables geometry-aware open-vocabulary grounding, associating arbitrary natural language queries with localized 3D regions and objects in dynamic environments. Unlike existing approaches that require calibrated, posed RGB-D input, RADIO-ViPE operates directly on raw monocular RGB video streams, requiring no prior camera intrinsics, depth sensors, or pose initialization. The system tightly couples multi-modal embeddings -- spanning vision and language -- derived from agglomerative foundation models (e.g., RADIO) with geometric scene information. This coupling takes place in initialization, optimization and factor graph connections to improve the consistency of the map from multiple modalities. The optimization is wrapped within adaptive robust kernels, designed to handle both actively moving objects and agent-displaced scene elements (e.g., furniture rearranged during ego-centric session). Experiments demonstrate that RADIO-ViPE achieves state-of-the-art results on the dynamic TUM-RGBD benchmark while maintaining competitive performance against offline open-vocabulary methods that rely on calibrated data and static scene assumptions. RADIO-ViPE bridges a critical gap in real-world deployment, enabling robust open-vocabulary semantic grounding for autonomous robotics and unconstrained in-the-wild video streams. Project page: https://be2rlab.github.io/radio_vipe

Problem

• open-vocabulary
• semantic SLAM
• dynamic environments
• monocular RGB
• multi-modal fusion

Innovation

• open-vocabulary SLAM
• multi-modal fusion
• monocular RGB video
• dynamic environment
• foundation models