3D-Aware VLMs with Implicit and Explicit Geometries

2026-07-23

AI Summary

Current vision-language models predominantly rely on 2D inputs, limiting their performance in tasks requiring fine-grained 3D spatial understanding. This work proposes VLM-IE3D, a novel framework that, for the first time, leverages only RGB videos to jointly incorporate implicit geometric priors and explicit geometric structures, thereby endowing the model with strong 3D inductive bias. The approach effectively integrates these two forms of 3D geometric representations with 2D visual features through Implicit Geometry Tokens (IGTs), Explicit Geometry Tokens (EGTs), and a 3D-aware adapter module. Extensive experiments demonstrate that VLM-IE3D significantly outperforms existing methods across a range of challenging tasks, including 3D video grounding, visual localization, dense captioning, and spatial reasoning.

Abstract

Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.

Problem

• 3D-aware VLMs
• spatial understanding
• 3D reasoning
• vision-language models
• 3D tasks

Innovation

• 3D-aware vision-language models
• implicit geometry
• explicit geometry
• geometric tokens
• RGB-only 3D understanding