Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

2026-09-29

AI Summary

This study addresses the challenge that multimodal large language models struggle to integrate multi-view images into a coherent three-dimensional spatial understanding. Inspired by human cognition, this work proposes an "imagine-then-answer" paradigm in which the model decodes multi-view images into 3D Gaussian Splatting representations via learnable summary tokens to imagine a coarse-grained scene. A photometric reconstruction loss is further introduced for joint training, implicitly enhancing cross-frame correspondences. The proposed approach consistently outperforms existing methods that rely on pixel-level geometric details across multiple spatial reasoning and 3D understanding benchmarks. These results demonstrate that explicit scene imagination is more effective than directly injecting geometric features, offering a promising direction for advancing the 3D reasoning capabilities of multimodal foundation models.

Abstract

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

Problem

• Multimodal Large Language Models
• 3D spatial reasoning
• multi-view images
• 3D understanding
• cross-view correspondence

Innovation

• Multimodal Large Language Models
• 3D Gaussian Splatting
• Spatial Reasoning
• Summary Tokens
• Photometric Reconstruction Loss