M3: 3D-Spatial MultiModal Memory

2025-03-20

AI Summary

This work addresses core challenges in 3D feature distillation—namely, high memory overhead from high-dimensional features, misalignment with base-model representations, and information loss—by proposing a 3D multimodal memory system tailored for medium-scale static scenes. Methodologically, it is the first to systematically tackle the 3D feature compression bottleneck via principal scene component decomposition and Gaussian memory attention, integrating 3D Gaussian splatting, vision-language models (VLMs), perception models, and large language/multimodal models (LLMs/LMMs) into an end-to-end differentiable training framework. The approach achieves significant improvements in feature similarity, downstream task performance (e.g., semantic segmentation, pose estimation), and visual feature tracking fidelity. It has been successfully deployed on a quadruped robot operating in indoor environments, demonstrating real-time inference capability and strong cross-scene generalization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

We present 3D Spatial MultiModal Memory (M3), a multimodal memory system designed to retain information about medium-sized static scenes through video sources for visual perception. By integrating 3D Gaussian Splatting techniques with foundation models, M3 builds a multimodal memory capable of rendering feature representations across granularities, encompassing a wide range of knowledge. In our exploration, we identify two key challenges in previous works on feature splatting: (1) computational constraints in storing high-dimensional features for each Gaussian primitive, and (2) misalignment or information loss between distilled features and foundation model features. To address these challenges, we propose M3 with key components of principal scene components and Gaussian memory attention, enabling efficient training and inference. To validate M3, we conduct comprehensive quantitative evaluations of feature similarity and downstream tasks, as well as qualitative visualizations to highlight the pixel trace of Gaussian memory attention. Our approach encompasses a diverse range of foundation models, including vision-language models (VLMs), perception models, and large multimodal and language models (LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy M3's feature field in indoor scenes on a quadruped robot. Notably, we claim that M3 is the first work to address the core compression challenges in 3D feature distillation.

Problem

• Address computational constraints in high-dimensional feature storage
• Resolve misalignment and information loss in feature distillation
• Enable efficient training and inference in 3D feature distillation

Innovation

• Integrates 3D Gaussian Splatting with foundation models
• Uses principal scene components and Gaussian memory attention
• Deploys feature field in indoor scenes on robots