SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

2026-08-05

AI Summary

Existing 3D scene understanding approaches rely on fixed multimodal combinations, which struggle to dynamically adapt modality usage according to the input query, often introducing semantic noise and limiting reasoning capabilities. To address this, this work proposes SmartMage, a unified multimodal large language model that introduces two key innovations: Semantic-guided Multimodal Adaptive Routing (SMART) and Modality-Aware Gating of Experts (MAGE). These mechanisms jointly enable adaptive activation of relevant modalities and expert modules based on query semantics, text-modality alignment, and modality quality. SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks and demonstrates strong results on RGB video tasks. Ablation studies using ScanFacet confirm the effectiveness of its semantic-modality matching strategy.

Abstract

Understanding 3D scenes is fundamental to embodied intelligence, requiring joint reasoning over heterogeneous information from multiple modalities, including visual and geometric cues. However, the relevance of these modalities often varies across queries. Existing Multimodal Large Language Models (MLLMs) typically rely on fixed modality combinations, overlooking query-dependent modality needs. Such a rigid design can introduce semantic noise from irrelevant modalities while underutilizing more informative ones, leading to wasted computation and diluted reasoning. To address these challenges, this paper proposes SmartMage, a unified MLLM that dynamically orchestrates heterogeneous modalities for semantic-aware 3D scene understanding. Specifically, SmartMage incorporates: (1) a Semantic-guided Modality Adaptive RouTng (SMART) module that selects task-relevant modalities using semantic priors, text-modality alignment, and modality quality; and (2) a Modality-Aware Gating Expert (MAGE) module that leverages modality priors to guide expert activation, fostering adaptive specialization in multimodal reasoning. Empirically, SmartMage achieves state-of-the-art performance across five 3D scene understanding benchmarks, and attains competitive results on RGB-only video understanding benchmarks. In our diagnostic benchmark ScanFacet, tasks are divided into fine-grained semantic categories, enabling analysis of modality combinations preferred by each semantic type. The observed modality-semantic patterns provide further evidence of SmartMage's effectiveness. Project page: https://yuecheong.github.io/SmartMage/.

Problem

• 3D scene understanding
• multimodal large language models
• dynamic modality selection
• semantic-aware reasoning
• modality orchestration

Innovation

• Dynamic Modality Orchestration
• Semantic-guided Routing
• Modality-Aware Gating
• 3D Scene Understanding
• Multimodal Large Language Models