Dynamic Resilient Spatio-Semantic Memory with Hybrid Localization for Mobile Manipulation

2026-05-30

AI Summary

This work addresses the challenges of geometric inconsistency, semantic incompleteness, and unbounded memory consumption in existing mobile operating systems within dynamic indoor environments. The authors propose a real-time, map-free framework that constructs a spatial-semantic voxel memory online from RGB-D observations, integrating LiDAR-inertial-visual SLAM with pose-graph optimization for robust perception and manipulation. Key innovations include a pose-graph-aware Redundant Memory Pruning (RMP) strategy that ensures efficient, bounded-memory updates following pose corrections, and a reliable object relocalization pipeline combining language-guided 3D retrieval, open-vocabulary detection, and multimodal large-model semantic verification. Experiments across four dynamic scenes demonstrate improved long-horizon task success rates—rising from 40%–60% to 55%–70%—with memory usage constrained to 0.37–0.63 GB and memory update latency as low as 0.43–0.53 seconds.

Abstract

Reliable mobile manipulation in dynamic indoor environments requires a scene representation that remains geometrically consistent, semantically queryable, and computationally bounded as the environment changes. Existing systems often rely on pre-built maps, static-scene assumptions, or highly accurate camera poses, which can lead to stale or misaligned scene information when target objects are relocated or pose estimates are corrected. This paper presents DREAM, a real-robot mobile manipulation framework that integrates perception, memory, localization, navigation, and manipulation in previously unseen indoor environments without a pre-built map. DREAM constructs an online spatio-semantic voxel memory from RGB-D observations registered by a LiDAR-inertial-visual SLAM backend. It further introduces pose-graph-aware Redundancy-Aware Memory Pruning (RMP) to update historical observations after pose corrections while keeping long-horizon observation history bounded. For target localization and reacquisition, DREAM combines language-conditioned 3D retrieval, open-vocabulary image detection, and multimodal large language model based semantic verification. Real-robot experiments in four dynamic indoor laboratory scenes show that DREAM improves long-horizon task success rates from 40%-60% with DynaMem to 55%-70%, while maintaining a memory footprint of 0.37-0.63 GB and an online memory-update time of 0.43-0.53 s across scenes.

Problem

• mobile manipulation
• dynamic environments
• spatio-semantic memory
• pose correction
• scene representation

Innovation

• spatio-semantic memory
• memory pruning
• hybrid localization
• mobile manipulation
• online scene representation